Agree with the don't bother comments
About the only decent spot is white haven beach at the very northern end (where all the poster shots are taken at full to half tide with clean water).
To do do this you will want the tides right and the wind right.
There is a good anchorage on the other side of the lookout so that is not an issue, (just have to row ashore and walk over the hill) but be warned all the tourist boats pull up at the same time so you could find about 200 other swimmers at the same place. but frankly you will spend all that time lugging the stuff around and only use it minimal time if all the stars align

And this is coming from someone who is super keen and out when ever it is windy

I took my gear becasue I was on a driving holiday from NSW so planned other spots along the way.
Carried the gear on a boat for 7 days and only used it once, there is plenty of other stuff to do