Cuttyhunk is really nice and unspoiled.
Minnie your experience with Fishers is one of the "insiders" as you alluded to. You spent time there growing up so I am sure the locals dont see you as an intruder. I can only tell you how we were treated and we were pleasent and took a positive approach.
We have been to many other places where the people are equally as wealthy such as Sag Harbor, Marthas Vineyard, Catalina Island California. Newport California, Bainbridge Island Seattle yet have a a much friendly attitude to those who travel there. Having money and wanting to keep the "common rift raft out" attitude dont always go hand in hand. I know other sailors also who have been treated similarly at Fishers Island. I like James Wilson was looking forward to visiting the island for many years. Been there now and done that.
Like I said we will avoid ( not that the they care) Fishers Island and go to Stonington when in that area. Stonington is where the Scallop fisherman are home based. They even have a very unique place to purchase scallops on the honor system and quite a quaint little town. I was amazed at all the 40+ HInkleys and Morris's moored at Dotsons where the mooorings are very close you so can brush your teeth in the reflection in the transom of the Hinkley moered in front of you.
Stonington Seafood Harvesters - Stonington, CT