You should read the articles in Latitudes & Attitudes magazine writen by Captain Woody about his cruises. He was your age when he left and isn't that much older now. I myself am 35 so I am not ancient. I think it is easier to go when you are young or retired, because both of these points in your life are about changing comittments so it is easier to make the "leap" and leave the dock. People who are of retirement age typically have more disposable income than people at age 25 so I think that would be the main reason that people go later in life as it makes it easier to purchase and outfit a boat. People your age who do go tend to go as crew for someone else.
Also some, but not all gradually work themselves up to going by amassing experience over time. I am one of those who is comfortable diving into a project and immersing myself in it. Most prefer to work up to something over time and gain experience. By the time you gain experience, you realize that you have a wife and kids who aren't into sailing (there are of course exceptions) and the dream gets put on hold until you retire and/or divorce :-).
One thing you will find as you get older is that time moves faster.
I find that my appreciation of the passage of time is now about half of what it was in my early 20s. Time just seems to move at twice the speed, oddly I am not any more busy than I was 10 years ago, but there are more distractions/responsibilities. When you are in your early 20s or retired there is a lot less distraction or rather you are in a better position to ignore it. It gives you the chance to sit back and think about what you really want to do, plan it, and implement it. Along those lines, my one piece of advice: Make sure you go! Every month/year that you put off leaving the dock the harder it becomes to go, until you realize one day that 10 years are gone.
FYI: I am envious!