Given you have a Goya Quad i'd say you are into wave sailing... despite what people say my experience is that if you sail year around you can expect anything from 5.7 to 4.0 weather. Not sure what your weight is but there are plenty of days with 40+ knts. The big problem is that those days don't regularly seem to line up with weekends. So if your work schedule is flexible you should be able to catch some really windy days / good swell.
Most commonly used sail sizes are 5.7 to 4.7... you will need a floaty board for those light days with good waves.
As someone suggested the shop to go to is Windsurf and Snow. They have some rental gear as well as a ton of used gear and previous season gear at reduced prices. The Seabreeze buy and sell is also a good source for the odd bargain.For wave sailing spots along the Sydney coast look at this guide:
nswwavesailing.org/spot-guide/The events are also a good way to meet other windsurfers that are into waves... or just have a beer with.
A lot of the slalom guys sailing Botany Bay don't have sails smaller than 5.4... but I have a feeling they just don't go sailing when the wind hits 25 knts. +This summer should be pretty good for NEer seabreeze if the El Nino forecast comes good.
Flights to Western Australia can be cheap if you book well in advance and Fiji / Hawaii / Indonesia / New Caledonia aren't too expensive either.
As Whyner suggests Gerroa 2 hours South is a pretty special place. When you have a NE seabreeze with Southerly swell it's probably one of the funnest places on the planet to wave sail.... especially since you don't have to deal with sharp coral reef.