Are you seeking cabin
size or cockpit
size? With the kids gone, you should figure out whether below-decks elbow room or pack-the-cockpit social sailing is more important for you. Because on boats under 25', you can only have one or the other.
We chose cockpit room on our SJ21 Mk1 -- it fits five adults, six if one or more occupies the foredeck. But below, it's strictly sitting headroom and 2-man tent sleeping arrangements. No privacy for the head, no galley, little stowage. A Cat22 has substantially better cabin space, but its cockpit is quite tiny. Likewise the Ensenada/Balboa 20.
Water ballast allows fairly large boats to be trailed behind small vehicles, but at some cost in initial stability.
Here's another bind on small boats: you want shallow draft for gunkholeing. How shallow? Pull-it-onto-the-beach shallow? Do you actually want to sail
in 2' of water, or just motor in and beach or anchor?
A swing keel boat will get you to the beach, tho some keels retract more than others. But most don't sail well (or at all) with the keel up. These boats often have large, awkward keel trunks inside the cabin. A shoal keel boat like the Newports or Compacs will let you sail in skinny water, but you pay for it in miserable upwind performance. Wing keels or medium shoal keels get you into 2 or 2.5' of water. Be aware wing keel + muddy bottom = Rocna anchor.
Other arrangements are keel-centerboard (not very common below 26') or lifting keel, like you find on some S2s. Some boats to look at:
Compacs, esp the catboats (expensive)
I append the Mac26D
to your list because, while not as roomy as the X or M and lacking a 50hp outboard, it is a genuine sailboat with sweet lines and a reputation for good handling. It sits very low on its trailer and is easy to launch and retrieve. It has an enclosed head, a queen aft berth, a bright (if plastic-y) interior, and decent headroom. It'll tow at around 3000 lbs all-up and costs half to a third what 26Xs are fetching.
I have some love for the 26D.