The link below is for places in 3 Mile Harbor:
East Hampton Cruising Guide by OspreysGuide.com
Of the places on that list I have been to Bostwick's (good), Jeneen's (ok) and Damark's Deli (good for a deli). I have never been to 'East Hampton Point' but found some discouraging reviews on the web. The other places on that list are a pretty good hike from any dock.
Like most Hampton locations this harbor can get busy and crowded during the summer. If you come here wanting lunch or dinner make sure not to overlap with the fireworks show that is in July (unless you want to see fireworks) as the harbor is full of boats then.
As for Sag Harbor (
Sag Harbor Cruising Guide: Restaurants ) it seems you can not tie up to the dock and must anchor and row in. They seem to have no problem allowing huge mega yachts tie up at the docks though.
I would think that Greenport would make a good destination for docking & dining. Check the ospreysguide link for info on this and other places.