In my experience the southerlies do literally come howling through. You can be cruising along in a nice 17 knot NE and have a very quick change to 30+knots S.
Keep an eye out to the south for building clouds but sometimes they come out of clear blue skys. Usually preceeded by a short calm period, but not always. If the NE wind starts to kick slightly N or NW come in immediately. The southery is not far behind.
If you remember the Kurnell disaster a while back you'll know what i mean.