Sailing to the South Pacific or Caribbean is a lot different than flying there and daysailing. I think there would be a lot more sense of accomplishment. If that is what you're dreaming of doing I say go for it. Do your research. Read a few books, Lin and Larry Pardey's "The Cost Conscious Cruiser" is a good choice and any others you can find. As far as a boat, search
New and Used Yachts for Sale - YachtWorld.com for boats that are available in various price ranges. A good cruiser can sail anywhere you wish to go. The debate between newer fin keel boats and solid heavier "old shoes" is one that will never be settled. Who knows, in 30 years these debates might be between an old heavy Farr design and something newer and lighter.
I live on a sailboat in Victoria harbour. Amongst my neighbours are a couple who've sailed around the Horn in a Trintella 49 and a couple with a Cape George 36 who've been around Vancouver Island and will be heading offshore in the future. Last fall another couple left for the South Pacific on a Perry designed Baba 30. Another neighbour sailed from Victoria through Panama and along the coast of South America to Brazil in a Haida 26 some years ago. He is currently nearing the Falklands after leaving Puerto Montt Chile and rounding Cape Horn in his Spencer 35, not unlike an Alberg 30. Lots of well found boats of different types can sail offshore and they all have advantages and involve compromises. The more you know the more apparent these will be.
After you buy a boat learn to sail around Victoria. It's a fantastic cruising area with many protected anchorages and easy access to offshore conditions only a few miles away. A lot of cruisers round Vancouver Island as a shakedown to the offshore trip. This lets you find out what doesn't work before it's too late.
Sailing to the South Pacific is something that you work up to but it's not impossible having been done by many successfully.