1) The buoy is there for a reason, primarily commercial shipping with deeper drafts
2) Prudent passage planning requires that buoys and other aids to navigation should be identified in advance of departure. There should be no surprises.
I would leave it to port.
ISPA Yachtmaster Offshore Instructor Evaluator
Sail Canada Advanced Cruising Instructor
IYT Yachtmaster Coastal Instructor
ASA 201, 203, 204, 205, 206, 214
As I sail, I praise God, and care not. (Luke Foxe)