If you must trailer, you definitely want a boat smaller than 25' and probably close to 20'. About 10 years ago I was in a similar position as you. I wanted a small trailerable boat, big enough four day sails for my family of 5. At the time my kids were 2, 5, 8. I did a lot of research and decided on a Catalina 22. I bought an early 80's model for $4500 with trailer, good sails, good outboard, etc.
For the first season we trailer sailed the boat. We would load the gear on the boat at home, pack up, trailer to the marina, rig
, launch, sail, recover, un-rig, and then drive home. To be honest, trailer sailing was a real drag, and took much of the fun out of sailing. Even with practice, it took a solid hour from the time we arrived at the marina until everyone was aboard and we were motoring away from the slip. And we had another hour at the end of the day - which was not fun at all when everyone was tired and cranky (me included).
For the second season purchased a mooring and left the boat there. That was SO much better. Not it took 15 minutes from when we arrived at the marina until we were underway, and just 15 minutes from when we arrived back at the marina.
A 22 boat was nicely sized for day sails. The cabin was big enough for the kids to go below when they got tired. The porta potty worked great, and there was plenty of room in the cockpit and on deck. We spent a lot of time anchored and swimming in the Long Island sound.
After 1 year I was hooked on sailing and wanted a bigger boat. The first guy who came and looked at the Catalina bought it - for $500 less than I paid.
IMHO you should be able to get a very nice 22ish sized boat for well under $5000. Looks for Catalina, O'day, Hunter, Beneteau, C&C, etc.