you can do it in a sunfish if you are good at weather forcasting. As far as comfort level, I don't think there is such a thing as "too large a boat"
each to there own.
Ever seen 20' waves up close from the cockpit of a 25' boat? can get pretty exciting. At 4-6 knots it takes alot of time to get to an inlet and out of the bad weather.
I just sailed from Central florida to upstate NY in a 32' it took me 2-3 months and was lucky to only get caught in 2-3 days outside in 10-12' rollers and breakers. Of course those were the days the fuel pick-up clogged , engine was out, and we were off the jersey coast. Fortunately we were able to enter Barnaget light inlet under sail and safely anchor to make repairs to the fuel system.
Sailing can get exciting at unexpected times. preparation is best completed before casting off.
The more I sail , the more I become a fair weather fan.