1) The buoy is there for a reason, primarily commercial shipping with deeper drafts
2) Prudent passage planning requires that buoys and other aids to navigation should be identified in advance of departure. There should be no surprises.
I would leave it to port.
Green returning! I would also have left it to port.