Most have covered the other things to do when in town, a couple I might add...
Check out the Red Red Wine Bar, a student of mine is now the chef, do the sample flight of wines, pretty fun.
Joss isn't what it used to be unfortunately, the last three times I have eaten there is was pricey and nothing special. Still good, but not great.
I like a greasy dive as much as anyone, but I cant do Chick & Ruth's again, just see to many things that make one gag.
As mentioned, Navel Academy Museum is awesome, Bacons off West St. is entertainment for at least 2 hours. I like to walk some of the local boat yards once and while, my favorite for brokerage boats is Bert Jabins
If you're in the area with a car, St Michaels is about an hour, check out the maritime museum. Baltimore can be fun, especially Canton & Fells Point are fun if you like night life. Some good restaurants, bars, and live music.