I've lived and sailed the Long Island Sound from the north shore my whole life and think that if you are coming out from the City by train and cost is a concern you should look at mooring in Port Jefferson Harbor
at the Setauket Yacht Club
Here's my thoughts:
By Rail: The best choices are Port Washington, Oyster Bay, Port Jefferson & Greenport. These are old shipbuilding towns that have the train very close to the harbor because it was used to move materials.
Cost: The farther East you go the less expensive. Waiting lists, residency requirements and cost/foot will go down the farther you are from the City. But the cost starts to go backup on the East End.
Location: Port Washington & Oyster Bay, easiest access to NYC, most expensive. Port Jeff, 1 1/2 hrs from Penn Station, least expensive. Greenport, 2 1/2 hrs from Penn, more expensive but access to Gardiners Bay too.
Port Jefferson is tourist town that has a ferry from CT with many restaurants. It's central location puts most of the Sound's harbors a daysail away. The Setauket Yacht Club is a "sailor's club" which is helpful for a new owner. They have very good launch service. The club facilities are clean and they have a good racing and teaching program.
BTW: I'm not a member of SYC, just an admirer. We used to be members, but left the club when we moved over to Mt Sinai Harbor. Also check out the Mt Sinai Sailing Association(link below) which is the sailing social club of the area. If you have any more Q's about this area just ask