Getting to the Dry Tortugas will be a pleasant downwind sail and there are plenty of anchorages once you are there. It would be a good place to explore for a couple of weeks.
But and it is a big but getting back to Key West will be a 70 mile beat into the wind unless you are very very lucky. I would not want to have to do it on a fixed schedule even on a biggish boat.
A Catalina 22 could make the trip esp. if you waited for a good weather window for the return but you could be waiting for weeks.