As a lifelong sailor, and more recently a recreational diver also, I second that emotion.
Please stay well clear (let's say 500 feet) of dive flags.
And at night, if you see a red-white-red vertical light combo, that boat can't maneuver out of your way. They may or may not be dive boats. So please 'go around'.
Some dive boats 'drift' over their divers-down, due to current that makes an anchored location impractical (like the reefs off S. Fla with the Gulf Stream current). So even though they're maneuverable, they can't leave their divers.
If for any reason (meaning navigational safety) you (and other passing boats) can't easily miss them, then radio the dive boat and tell them the situation so they can move if necessary. If you work with each other rather than just cussing out each other's location and sailing/powering away mad, you can make it work. Some of those divers or boat captains are sailors, they'll know what's up.
Here's a new Public Service Announcement on the topic:
YouTube - Dive Flag Awareness Public Service Announcement