A well-found 37-42' boat will get you from one side of the world to the other, but be far easier to handle short-handed or in the case of equipment failure. IMHO, one of the biggest source of cruising couples failing is that they get more boat than they can comfortably handle short-handed. Most cruising couples are effectively two people who are sailing single-handed a majority of the time, with only occasional help from the "crew"—since the not-actively sailing "crew" will often be busy sleeping, doing navigation work, cooking, watching and caring for the baby.... and so on.
Beth Leonard, author of "The Voyager's Handbook
" and Evans Starzinger sailed on a 37' boat, and have circumnavigated in it and then some.
Some other points to consider:
A smaller boat will be easier to dock and handle in the harbors and marinas.
A smaller boat will be less expensive to maintain, haul, dock, store, insure and own in general.
A smaller boat will be a less tempting target to pirates and thieves.
A smaller boat often has a shallower draft and can often go more places than a larger boat.
A smaller boat often has less freeboard than a larger boat, making MOB recoveries simpler.
A smaller boat has gear that is easier to handle—ever check out the anchor on a 50' boat???