We just went from 33 to 41 feet, and that was the limit I thought my wife could single-hand in terms of actually reaching around a furled boom, etc. and to a certain extent anchoring (we have 3/8" chain and we prefer to manually operate the windlass).
If you are both fit, 40 and over 5' 9", you two could probably handle a ketch...with a lot of practice and if you sail conservatively re: reefing early and taking full advantage of the split rig. But you have to visualize boat handling alone, at night, in 40 knots when a squall lays you down, your wife is on deck on dog watch, and you wake up to the sound of your kid screaming because she's wriggled in her sleep halfway out of her lee clothes and the boat's pitching has thrown her out of her bunk. Her arm may be broken...you can't tell because you can't get to a light because you are holding her against the motion of the boat to prevent further injury. You can't get on deck, and you can't TELL your wife you can't get on deck because the wind is howling and she's busy trying to crawl to the traveller to let fly the sails so the boat isn't pressed over.
Then a halyard parts. Over that twang, something goes "clunk" in the vincinity of the engine mounts. What was it? Is it safe to start the engine? Are there lines in the water? Is my wife even still aboard? Was she clipped on? Why won't my kid stop screaming?
Perhaps this description is overly dramatic, but you can certainly get it ten times a summer on even little Lake Ontario. If you are very experienced, even an older guy like Hal Roth (who can and did sail a 50 foot racer-cruiser solo for a number of years...and then went to a well-founded 35 footer for ease of use) has an answer and an action for most of those dilemmas, but "learning via baptism of fire" isn't an option when kids are aboard.
We plan extended cruising, possibly a circ, in a 41' steel cutter, but we are deliberately waiting until our son is almost 8 to go (he's 6 at the end of this summer), because his ability to take care of himself increases each year, as does his ability to keep one hand for the boat and to understand the physics of sailing and to develop "sea legs". We are going to spend a winter aboard as well in order to REALLY get used to "life aboard" under trying, if static, conditions. That's more a baptism of ice cubes here in Toronto, but the kid can handle the icy docks and can walk to school, at least.
By all means follow that dream. But evaluate critically the need for a bigger boat vs. the capabilities of ALL crew to handle living aboard and working the boat in all conditions.