What pschoonveld said.
I'll add:
The maximum current in the NYC area is about 5 knots in Hell Gate and off Roosevelt Island on the East River - and that is at peak flow. Everywhere else is much lower but can slow you down considerably if you are against the current.
The words 'cheap' and 'dockage' are not usually used in the same sentence when describing marinas in NYC. On the cheap side would be the 79th St. Boat Basin where a mooring is $30/night:
Marinas : West 79th Street Boat Basin : New York City Department of Parks & Recreation : NYC Parks
I've heard that it can be a bit bouncy there in some wind conditions if you are considering sleeping aboard.
NYC Parks also operates another mooring field down by Houston St. just north of the N. Cove Marina
Hudson River Park - Moorings
I'm pretty sure they try to discourage transients but it is worth a try. The moorings down here are a little better protected then up at 79th St.
Liberty Landing Marina has a
fuel dock and
pump out:
Home - Liberty Landing Marina
This marina is well protected but it ain't cheap as previously noted. I would stay here at least 1 night as it is a pretty nice location. It will be much quieter during the week then on weekends.
As for actually sailing in NY Harbor I find it gets to be a bit trying with all of the commercial traffic. The view is great though.
There are a few clubs in Raritan Bay as suggested. Not sure if they offer transient moorings.
Home - Liberty Landing Marina
Search for 'Highlands marina nj' and you will find a few other places on Raritan Bay.
25 nm. up the Hudson is Nyack Boat Club which has cheaper transient moorings. Transient dockage in Tarrytown should be cheaper then NYC.