You could purchase an older 1970's era boat for around $10K to $15K with money left over to add other options, if needed. I agree with the above comment on buying a larger boat now to avoid having to upgrade very shortly.
Try a 27 ft. Catalina, Ranger,San Juan, Cal, S2, or others. They sail well and are easily single-handled. You will find that a trailer-sailer boat gets tiring each and every time you have to put the mast & rigging up & down before the days end. It is much more enjoyable for you and crew to merely arrive at the marina and board yourselves and supplies for a fun day of sailing. Bu, if you don't believe you will be sailing much each year - buy one that is trailerable.
Ask someone pulling a sailboat how often they get down to sail their boat.
Best of luck...