I will add myself to the list of those who will not set foot in Chick and Ruth's or Cantler's. Chick and Ruth's food is way too salty and poorly prepared. I used to be a local and would steer everyone away from there. Cantler's is not worth the trip even by water. Unless you really like crabs and are willing to pay through the nose for them (while there are plenty of other more reasonably priced places nearby) I would not go. I used to crew on a charter boat that did Cantler's trips. Usually they guests would pay for the crews meal there. I brought my own food.
As for good restaurants I will +1 O'Learys. Expensive, but the food and service are outstanding. Typically the further form Ego Alley you get the better the food. +1 to Annapolis Iee Cream up Main St. Try their peanut butter oreo.
Fawcett's is a great chandlery. Since they moved out of downtown their prices have become very reasonable. You will find the staff experienced and knowledgeable. I still buy from their online store (pyacht.com) because many times the prices are better than others.
If you want to kayak while there check out Annapolis Canoe and Kayak (aka Spring River Kayak). The staff is great. As Dave recommended, take a sail on the Woodwind. Especially if the wind is in the upper teens. Nothing like sailing on a 70ft schooner with the leeward rail buried. If you go tell the captain Nick & Renee said hello.
The Naval Academy never gets old. A must see is the model ship museum as mentioned above.
It's only been 8 months since I left and I really miss living there.