I've been to the Caribbean and the Bahamas in the summer. My wife prefers the summer weather, which is more calm and the water even warmer. We've flown down each time, never aboard the boat. Always had an escape route that had us back in New England the next day. Big difference from a sailboat.
However, watching tropical weather developments from my comfy protected couch can be exhausting. It would be a real buzz kill to worry about being bug out ready all the time on the boat. I'm recalling when I decided to change fuel filters at anchor once and the filter house accidentally broke. I was SOL for a couple of days, trying to source the repair. I think I'd be constantly worried about events like that.
I'm not sure the absence of crowds does anything to overcome it. In fact, I wonder if the busy seasons are just safer and more fun. There definitely places on this planet, where I'd rather have folks around than be fully alone and isolated. There comes a point, when no activity is boring too. I go back and forth, like energy, then like solitude. Even in the notoriously busy BVIs, no winter crowd has ever been in the way of us having a great time. nor kept us from going anywhere we chose.