In a previous life I was "stuck" in Albany for a few days sight seeing with the Ex
We stayed at the Emu Beach Caravan Park (other side of the Bay)
I sailed from the beach in front on the Caravan park and left my gear rigged next to my tent

It's cross onshore with a nice rolling swell (fetch?)
I found out later that a seriously big bitey was caught just off Midleton Beach (1992)
You can also sail at Frenchmans Bay, good flat water stuff, however you need big B** as this is also a known haunt of big fish that like to hang around old Whaling stations

Very pictureaskew piece of Coastline