Luckyp - Pt Fairy has some good spots right in the town. There's plenty of room on East Beach if you like to have fun in some good beachies with the family. Friday morning & Sunday look ok for most locations although the onshore wind on Saturday might prove to be a good time to sit back in the local or spend time on dry land with the family. From the mouth of the river around to South beach is a bit more serious and although I'm not overly familiar with that area, if you're competent then you shouldn't end up on the rocks.
I can understand LateStarter's comments - however Pt Fairy is a small town that relies heavily on tourism (no industry) and a prime holiday spot for Viccos that booms over Summer up until Easter, when it starts to settle-down a bit. I think you'll find that all the locals will be expecting the tourists...and happy to let them have a few waves over the weekend as it's a big boost to the economy. Once Winter starts to approach, the swell gets better and the tourists start to thin out, or get thinned out!
Just a note though - if you head any further West of Pt Fairy though...don't expect any niceties as the locals aren't as obliging. I'm just giving you a heads-up as I'm sure you know what to expect!