Since you both enjoy sailing, and seem quite confident that it's something you'll want to continue with your family, AND if you can swing the money I'd be very tempted to go for the 27 or something in that range. Your small tots will grow quickly, and this is an excellent time to start sailing with them.
I think you'll be much more comfortable with a larger boat and more inclined to spend extended vacations afloat with a family of 4 if you go to a 27/28 footer. A Cat 27, C&C 27, etc.. look at Rangers too perhaps, you should be able to find a decent example under $10K nowadays.
1984 Fast/Nicholson 345 "FastForward"
".. there is much you could do at sea with common sense.. and very little you could do without it.."
Capt G E Ericson (from "The Cruel Sea" by Nicholas Monsarrat)