PWC, I grew up not far from there, over by Glen Landing Middle School.
Each state's rules are different, but the boat will probably have to be registered in the state in which the boat is "garaged". That is, the state in which the boat resides, which isn't necessarily the state in which the owner resides. For example, I now live in PA, but my boat will be registered in NJ because, under NJ's laws, if the boat is there more than 180 consecutive days, it has to be registered there. Sure, I could try to "cheat" and go for a cruise offshore or up to NY/down to DE twice a year, but in the end it's easier to just pay the fee.
Some of the problems with sailing at the NJ shore include:
many of the inlets can be very difficult to navigate in a slow-moving sailboat;
most of the bays are small; and
there are LOTS of powerboats who don't understand your operational and navigational issues.
We kept our last boat in Great Egg Bay, which is behind Ocean City. We were in the back bay, and at low tide we would run aground (the back bay gets to 1-2', and sometimes less). Cape May isn't bad, but you'd be committing to sailing in Delaware Bay and the ocean. Do a little reading here and you'll see that most people (Ulladh being a possible exception) don't think too highly of Delaware Bay as a sailing destination. So, you're really left with sailing in the ocean. Probably not a big deal, but in my case I have a family with 2 kids, and the idea of sailing in a protected bay was more attractive than sailing in the ocean, especially when just starting out.
I think your best bet would be to figure out where you want to live, then buy a boat. The cost to move even a moderate-size boat several hundred miles is crazy ($3-5K easily; my move of my 25' Catalina from Toms River to Ocean City cost $500). Plus, the conditions will vary from location to location. So, a boat that works well in one place may really stink for another. For example, deep draft boats are, generally, more stable and faster than their shallow draft cousins. But a boat with a 6' draft, however, would have a heck of a time doing any sailing in most of the NJ bays. You could probably navigate the ICW (or what's left of it), but most of the rest of the bays would be off-limits. The same would be true in parts of Florida, for example. A deep-draft boat will keep you from exploring many of the inlets and other areas. By contrast, if you were to go to Maine, or the southern parts of the Chesapeake, the water is deeper there and the deep-keel boat might be just fine (unless you wanted to explore the northern Chesapeake).
As to boat insurance, the real question is why wouldn't you have it? If your boat comes loose from its dock/mooring in a bad storm and crashes into someone else's $750,000 yacht, do you really want to have to pay for the repairs out of pocket? Incidentally, that's why most marinas want it.
Ultimately, you said your goal is to figure out "how to get registration numbers on [your] boat, while still legally being an out of stater". Realistically, that's a piece of cake. You don't have to live in the state to register and keep the boat there. There may be hoops you have to jump through (such as getting the USCG documentation transferred into your name, if your boat is USCG documented), but it really isn't that bad.