I would suggest not to go smaller than 30', the bay can get ugly quick. On the other hand if you go over 35-40' you may find it very expensive and difficult to find a slip. Annapolis area marinas are outrageous, call a few and you'll see. Be weary of marinas that seem to good to be true.
One thing you definitely want to consider is draft, one of the main reasons we selected the centerboard configuration on our boat (4' board up). The average depth of the bay is about 5', if you enjoy gunk-holing you'll want to keep the draft under 5' if you really want to get into some of the prettiest and peaceful coves on the bay.
Otherwise, pick the one you fall in love with. We almost purchased a Pearson 30 (a great boat) that was considerably less, but when we saw the Tartan 37 we couldn't look at anything else. It just felt right!
Send me a email if you like.