There's a friendly crew of foilers who launch from the north end of Collingwood Beach, just south of Moona Creek in Vincentia.
This spot has good wind, good access and you can cover the whole Bay from here if you like.
The swell is variable in the bay depending on the direction, but the north east wind chop out in the bay can be good from this location as there's a long fetch for it to build some size in a decent wind.
I've seen Anyboard, just about, very nearly, almost paddle into a long downwinder on his downwind SUP

And I've also seen Anyboard, trailing the wing most of the way, just ride the wind chop from out of sight upwind to just about out of sight downwind.
Yeah, Jervis Bay is nice.