The only anchorage I know is at extreme south end of Liberty State Park in New Jersey. You go west from the harbor through the channel that leads to the Statue of Liberty (just to the south of the island). When you get to the ferry landing you keep going west through a narrow, marked channel to a small anchorage behind a ruined stone breakwater (might be a bit rolly with strong south winds but basically very well-protected otherwise).
There is a dinghy dock there (lock your dinghy well!) and you can walk to the New Jersey light rail transit stop in about 10 minutes. Less than 10 minutes on the light rail will bring you to one of the two stops of the PATH subway that goes into Manhattan (one branch to the World Trade Center; the other to 33rd St and Broadway). About $4 each way for both transit lines. You could also stay at Liberty Landing Marina which is at the north side of the same park - it has good protection from wakes in the river (almost no where else will you find this), but last time I checked transient dockage was $4.25 a foot.
Heading back to Lake Ontario for this summer. Relatively few stops along the way from Grenada. Martinique, Guadeloupe, St. Martin (must have something to do with the French food), then Bermuda, New England and up the Hudson/Erie Canal. We were going to go via Newfoundland and Labrador but June remembered that one of the kids is getting married this summer - details, details!