Welcome to the world of the new boater...they are far more important than you or I. Recently we were waiting in line, 3 of us, to get in to a local fuel dock/fairway to slip our boat. All 3 waiting patiently, when a 40+ ft Hunter just blasts by us all, screaming "give way" the whole time as they abruptly cut in front of everyone...no emergency, no failures - they just did not want to wait. She told me later...."our boat doesn't wait on you all"...Was all I could do to keep my mouth shut. Luckily they were not there this year.
when we lived in Annapolis in the 80's you were either in line at the bridge and out of the harbor by 9...or as you found out, it is like the wild west..and that was then...about 4 or 5 years ago I was up and it was as you described all weekend. It seems the anchorage is just a short cut, or more buoys to race around to most there..
We now go only during the week.
Best of luck