Couple notes. First, the thread is nearly 10 years old, but wanted to make sure the info was correct.
Atlantic Highlands Yacht Club (ahyc.net) offers moorings with launch service to non-members for $50/night. The town also has a marina and I believe the rates are under $3/ft for transients.
You may also
anchor inside the breakwater, outside the marked channel. New markers were installed in 2011 as many cruisers were anchoring in the channel. The club will provide launch service for $35/night.
The town is a regular stop before jumps south, to Bermuda or across the Atlantic due to the ease of resupplying. There is a small, but well stocked chandlery, auto parts store and large supermarket all within walking distance. The restaurants (particularly the Flaky Tart bakery) make it a nice weekend destination as well.