I second this suggestion. A stroll around the grounds, too, if not too hot for you. We've visited three times and still haven't seen everything.
And don't forget John Paul Jones tomb at the Naval Academy.
For food, there's almost too many places to mention:
Cantlers for Crabs and a riverside view might be a good stop
Cantler's Riverside Inn | Where the Watermen Gather
For Steak it's got to be Lewnes' Lewnes' Steakhouse
I like the Raw bar at Mc Garvey's
There's often a good band and Middletons.
Breakfast at Chick and Ruth's Annapolis, Maryland Delicatessen Chick and Ruth's Delly
is an experience.
Though, I like the Cafe Normandie brunch.