That is a valid point and the seas can really whip up if the wind opposes the tide/current at the mouth of the Potomac. Even on a nice sailing day its likely to be pretty rough there compared to the rest of the Bay.There are a couple marinas on Smith Creek (first creek on the MD side on the Potomac River) The River is quite wide there and the sailing is fine.
I would note that this areas is controlled by 2 different weather patterns often (one coming down the Potomac and the other from The Bay). This can cause havoc in the area. As an Annapolis area sailor who has sailed there on many occasions, I can attest that the worst storm I have ever sailed through was as one squall came down the Potomac and the other squall came down The Bay and joined forces at Smith Point (VA side), waves were about 8' and it was severe. It started about 4 in the afternoon and lasted until several hours.