Don't know of any spots in Warrnambool itself but just down the road at Port Fairy has some great spots. Either at East beach or up the road at Killarney. I spent Christmas there and had some great days.
I meet a guy last weekend at Cape bridgewater who had just come back from a couple of weeks at Fairy port.
anyway eastern beach is it? the one in front of Golf course? that's good fun. and further back towards Warn. there was a boat ramp in front of a Caravan park, that looked OK. then somewhere around there was a spot where the rock groins created a mini harbour and flat water.