Last Saturday on San Francisco Bay the Singlehanded Sailing Society held their annual 3 Bridge Fiasco Race that you enter either single or double handed. It is indeed a fiasco because it is a pursuit race which means the start is staggered according to your rating and you can cross the line in either direction, round the marks that represent the 3 major bridges of the bay in either direction or on either side and this year there were about 350 boats entered. Fiasco indeed!
My point is I did this race singlehanded on my Beneteau 42 and had a blast. You don't do things fast, you must anticipate way in advance, you are really engaged thinking what if this what if that and what would I do if and you get a great workout!
The main thing is to stay safe and do not put yourself in situations that alone you can not handle.
That being said I believe there are tons of sailors out there that single hand even with others on board. "Single handing with wife (mate)" is the common term!
I am very close to your age and I really enjoy a great group of friends aboard and sharing the experience but for the pure pure pleasure of it nothing beats being alone, in tune with the wind and the boat, perched on the lee rail with one hand on the wheel having a great conversation with all the forces involved and grinning like a 12 year old Christmas morning.
P.S. Check out this photo taken on Vallejo Race last year and made of cover of Latitude 38. Not exactly grinning but perched on the lee rail and loving it.