Most of the harbors are on the CT side of the Sound. But LI has Mattituck, Port Jeff, Northport, Huntington, Oyster Bay Cold Springs, Manhasset Bay all of which are good anchorages and pretty easy to navigate into.
The CT side has more channels, rocky ledges, shoaling and bigger cities like New London, Bridgeport, New Haven, Norwalk. But there are some lovely smaller spots like the Thimbles, Guilford, Branford, Milford, Saybrook, Essex, Mystic, Stonnigton. The west end is narrow and has lots of traffic going every which way
If you pass thru Plum Gut you have Gardiner's Bay which has Sag Harbor, Greenport and several anchorages on Shelter Island.
You definitely can do several summers of weekends and hit some lovely stops, but New London is not central to these destinations. Pick up a Cruising Guide and figure out a good central location to moor the yacht. Depending on slips and moorings is not only expensive, but can be difficult. Count on anchoring and you have more flexibility and more cash on hand!
Your trips also will depend on your range. How much time do you want to sail each say and how far can you go?
And never forget that you are sailing in the land of "NO!"