I might be wrong here, and I'm not trying to seem adamant, I just don't want to invest in a boat only to turnaround in 3 years and try to flip it for a larger boat.
There is real logic to what you say. A boat is something that you are likely to want to pour your heart and soul into fixing up. Even if you do all the work yourself and buy your parts at home depot and swap meets instead of West Marine, boats do still have a way of eating up all your extra cash. So I can see wanting to buy only one.
On the other hand an absolute truism is that the smaller the boat the more you will take it out. What you need is experience and if your boat is easy to take out and sail for an afternoon, or a quick overnighter you will. The 31 footer I started on was a bit big for me and I didn't go out very often. When I did, it was for longer trips, but if I had a catalina 27 or a pacific seacraft 25, I'd be out every sunny afternoon.
Another important point is that you don't really know what you want in your offshore boat. Boat designs vary greatly especially in their undersides. Underwater profile matters. A modern boat is more easily driven in light air, is often "faster" and is much easier to park in a marina. A full keeled boat is more kind to it's crew offshore, and better in a storm, but is "slower" and maneuvering in a marina is so difficult it will make you want to anchor out. These differences and choices affect everything about your boat from it's sailing ability, to its complexity, to its cost, to it's interior etc etc etc.
You need to find out what kind of sailing you will actually do. It's nice to say "I'll always rely on sail" but it's another thing to find out that you actually will motor because you're bored to tears in light air. That's fine, but you'll hate yourself if you've sacrificed accommodations and strength for light air sailing ability and now you never sail in light airs. You might also find that a big, heavy, safe, storm-ready boat is so slow you hate it's sailing characteristics and regret all the emphasis you've put into strength over performance.
These preferences matter. The boats that suit them are very different. Until you know your preferences, and make your own choices, you won't know what kind of boat you really want. Your first boat (like your first girlfriend) is not likely to be the best long-term match.
Also, there is the budget issue. You "could" buy a 36 footer for less than 9K but if she has ocean potential, she'll be in awful shape and she is NOT going to be in a condition where you would likely be sailing her much. If your plan was to only live on her and work on her and sail other peoples boats (like club racing) that would be fine, but there is NO substitute for the experience gained in sailing your OWN boat.
If you choose one from the book I recommended you will have a boat you can afford, one that you can take for a sail at the drop of a hat, AND if you find you're okay with small, you don't HAVE to upgrade because all the boats in the book can (and have done) go around the world. So it could end up being the only boat you need to buy. You need a boat YOU can sail NOW and often.
P.S. Don't buy a wooden (or cement) boat. Just don't. Not okay. Never. I speak as a recovering wooden boat owner.