Greenport would be nice. There are a number of marinas with slips but I'm not aware of any moorings. I stayed at the Townsend Manor Inn, which was very nice and has a great big pool that my kids loved. The town has a new marina, but I have heard it is 'rolly' because of the shelter island ferry.
I have eaten at a nice restaurant called the Front Street Station, but I haven't been there in a year or so.
In Milford I have been to Milford Landing a few times. It's always been great. Good dockhands and free newspaper. There are lots and lots of restaurants in Milford, but I can never remember their names.
You really should look into the active captain cruising guidebook for some great advice.