I have a steel boat of fairly deep draft (5.5+ft). Because of maintenance issues and resale I do not think steel hulls are for everyone but they do have many many positive points. Alum however may be for everyone. I favor something like 'Strongall' aluminum constuction by Fairmetal Boats and others.
As far as fiberglass goes take a look at Airex foam cored hull and the stories are amazing. Pounded on rocks for hours, locked in ice floes for a month and they all sailed away with scratches (but deep scratches). Even modern wood/epoxy is wonderfully confidence inspiring. Take a look at Covery Island Boatworks and the kind of boats they are building and who they are building them for, folks that "know what it takes".
Pick your style of hull and then look for the best construction in that type of hull. Jimmy Cornell who writes about circumnavigation and organinzes ocean spanning rallies has a SHALLOW DRAFT aluminum hull (may well be a 'Strongall' hull). Consider that it is harder every day to find a spot to
anchor and the deeper the draft the harder it is. Also, the finest places to hang out and safest places to be usually require shallow draft. A boat that can stand up on the hard will pay you dividends many many times as well as being much much easier to find a place to store it when you have to leave it for some reason. A shallow draft cat can be run up on the beach if for some reason you had to bail out, big plus. Afterward the boat will likely be ready to go again. The cat can be beached nearly anywhere and easily (relatively) pulled up for repairs or storage. Look on Google Earth and you will see many interesting places around the world with no harbors. What if you could pick your weather and pull (plank/rollers/block&tackle) the boat above the tide
line and live aboard in your beach cottage until you were ready to go again.
If I can figure out a way I would like a 40' Wharram Cat someday. Shannon makes a boat called the 'ShoalSailer' that is very interesting. It is fairly new and so there is little in the way of a used market.
Be careful about buying anything too new. There are many many many older boats that people have poured buckets and buckets of money into only to figure out that their plans had changed and they need to sell. Find one of those boats. Don't get caught up in saving $ and getting a fixer upper because they can eat up years and many 10's of thousands of dollars and unless your goal is fixing up, then spending a big chunk of your life doing the wrong thing. For some reason the fixing up almost NEVER takes months, it always takes years. The exception being someone who lives on the water with a LARGE multi disiplinary shop and years of experiance fixing stuff or Bill Gates who can write a check and say "I want it yesterday".
Lot's of choices and fortunatly most of them are good ones, just different.