The only anchorage I know is at extreme south end of Liberty State Park in New Jersey. You go west from the harbor through the channel that leads to the Statue of Liberty (just to the south of the island). When you get to the ferry landing you keep going west through a narrow, marked channel to a small anchorage behind a ruined stone breakwater (might be a bit rolly with strong south winds but basically very well-protected otherwise).
There is a dinghy dock there (lock your dinghy well!) and you can walk to the New Jersey light rail transit stop in about 10 minutes. Less than 10 minutes on the light rail will bring you to one of the two stops of the PATH subway that goes into Manhattan (one branch to the World Trade Center; the other to 33rd St and Broadway). About $4 each way for both transit lines. You could also stay at Liberty Landing Marina which is at the north side of the same park - it has good protection from wakes in the river (almost no where else will you find this), but last time I checked transient dockage was $4.25 a foot.
Finished the circumnavigation in early February in Grenada. Have to work on a book project for the next several months so the boat will be waiting for next year.