One thing to consider there is more than twice the boat when you double the size. So there is a big difference between a 50 footer and 100 footer, although I have only been on one boat bigger than 45'. Just a guess on my part but parts for a 50 footer would be 4 times the cost of a 25 foot boat.
A 70' 160k fixer upper might take another 200k to fix up and a few years. Also a newbie crew or most any crew will not be able to handle a 70' boat. You might want to get started on the family to populate the boat and raising a crew.
In all serious though sounds like you have the money, time and probably knowledge to make this happen. If I had $150k to spend to see the world on a boat I would buy this
1990 Pacific Seacraft Crealock 37 Sail Boat For Sale - www.yachtworld.com
and go now. Or if you just want to explore the coast for now buy this.
Springline Yacht Sales, LLC (Mystic, CT)
and go now. You can't go wrong with a boat built by Pacific Seacraft, although there are plenty of others in this style to choose from, I just really like PS boats. Don't buy the boat for your eventual family because you will have to find or hire crew to sail it. You could maybe also go to boats in the mid 40' range. At 38' and above consider yawls, ketches or cutter rigs to break down the sails into smaller bite size pieces. More sails means each sail is smaller and more manageable for a small crew.
Go small go now.