Dodging racing fleets is standard getting in and out of Annapolis on a weekend. There are usually at least three or four fleets racing. I try to stay heads up and out of the way - they are racing and I am just out for a sail. I know that I would hate to lose a race because of a boat moving thru the fleet.
I use the same philosophy with working boats in situations where I have right of way. I figure that they are trying to make a living and I am out having fun. I maneuver early to avoid them so there is no confusion.