East Hampton Point is great on atmosphere and view, not bad on food, and pretty pricey. They have both a marina (also $$$$) and a dinghy dock which I believe is free while dining, but you had better confirm. Bostwick's is located at Harbor Marina, and is not bad. Just behind East Hampton Point is Harbor Bistro, which also has a nice view and has been excellent in years past. Michael's of Maidstone is a bit further away, and more of a townie hangout, as well as a bit more reasonable in price. Restaurants in the area tend to have a big turnover in normal years, and with the recession we need to wait until the seasonal places reopen to see what is good this year. Of course there are lots of good restaurants an easy cab ride away.
While we are talking restaurants one can get to from the boat, Sag Harbor has a huge variety an easy walk from the dinghy dock. On Shelter Island the Ram's Head Inn in Coecles Harbor used to be quite good, we will have to see what happens this year. Coecles is also one of the prettiest harbors anywhere, though you will have to watch your draft, especially at the entrance.
The Osprey Guide has been a great resource for the area, but last I checked it did not look like it was being kept very up to date.