Don't be discouraged. A touch of level-headedness helps bring those dreams to reality - otherwise, they are just dreams. You need to look at this with good judgement; financials, seamanship, practicality etc. and then plan accordingly.
There are various ways to achieve your dreams. I understand you aren't wanting to live in luxury and I didn't suggest how much money you'd need to spend to reach the minimum - others here are more qualified to do that than myself. And it is technically possible to do it on a little budget. This documentary has been posted several times ( Hold Fast - Travel Movies on Vimeo
) and shows what you can do with nothing. THEY did it. I wouldn't recommend that you follow in their footsteps; there is just so much in that video that I would consider unsafe. However, they weren't discouraged by cost.
There are many people that dream about just sailing away. With no debt and a portable skillset, your chances of doing that are much higher than the regular joe.
My advice would be to save even harder than you are doing now. Live as cheaply as you can and put more money into the boat; A safer, more comfortable small boat will be better than a cheap boat any day. You might "save longer" as well, spend more time learning to sale while you save - Feed your craving can stave off that need to leave and will better prepare you for your own boat. Start by sailing on other boats so that you know what you like and don't like before you buy; Buying the wrong boat with the wrong gear is an expensive lesson.