I think sundays forecast is for W-SW if it ends up being a westerly Ocean Grove/Barwon heads area can be a bit gusty as the wind comes over the headlands.....
Depending on how big you want the waves I would suggest heading to Flinders ( Big lefts, Cyrils, Hooters etc........ ) Portsea to Gunamatta will have very clean wind but with a 3m swell predicted it will probably be closing out, this is when the reef breaks of Flinders are probably at their best.

Phillip island - you can try CAt Bay, Right Point and Flynns Reef ....CAt bay is usually about 2 ft smaller than Right P and Flynns Reef. With the 3m swell running I would expect Flynns Reef to be approx 1.5m - 2m as it is inside Westernport.
If you are really competent you can try your luck at Express point but again the headland can play havoc with the winds.

All in all if you wake up Sunday morning and the wind is more SW than West I would head down towards BAwon HEads / Ocean grove if it is more west consider the above options.

Good luck....... I would put my hand up for a trip but I am hanging plaster in a mates house this weekend.