My first boat was a Bristol 22; at the time I had kids aged 5, 4 and 2.
My wife was not a super-confident sailor, but a great mate.
Here's my experience:
a) The kids LOVE IT. They are incredibly enthusiastic about all things boats and boating, and it's fun to be part of. The two year old started out on watch duty, looking for lobster pots, and the five year old learned to steer her first summer on the boat.
b) You'll need to figure out some rules that work for you all: ours were: cockpit or below only, always below during tricky maneuvers, serious unquestioning obedience to the skipper (bears repeating with children), PFDs in the cockpit at all times, no futzing around on or near the
ladder -- up or down only.
c) If it's family-only sailing, you're going to want to be able to singlehand the boat if you have to, and possibly just when the weather is getting bad. Ideally, your wife will be able to as well. Which brings me to my main advice:
Enjoy the Hunter, but get a friend who can sail with your family; that will reduce stress and increase safety in a major way. Both you and your wife should log some hours and classes alone, without kids, and get comfortable with each other before you introduce kids into the picture.
If you're going to be fighting over the provenance of your children while underway and in a pinch, you're seriously endangering your family. Like, heavy winds, serious roll, sails aren't reefed yet, rain and hail coming, two kids down below vomiting -- you both need to be on point and kicking ass above decks, and ignoring the kids if one of you can't take care of all that alone. So, don't do put yourself in that situation! Get settled and comfortable together first, add a third person if need be, and enjoy yourselves!
p.s. See if you can get that Hunter rigged for singlehanding.