It depends on the boat's use.
Most racing today is done on short windward-leeward, round-the-buoys courses which emphasize tactics, crew work and boat handling skills, so the shortest course is almost directly upwind half the distance and almost directly downwind half the other distance. Upwind ability is very important since velocity made good upwind will determine how the fleet places on half the distance of the course. Now that racing boats can plane downwind and achieve much greater speed downwind than upwind, however, a fast downwind boat can still overcome a slight disadvantage upwind.
Upwind ability is less important while cruising. Some cruising sailors eschew upwind sailing completely, "Gentlemen don't sail to weather", and will inevitably motor to windward in almost all conditions. If you cruise any significant amount, it is not unusual to go out on a perfect sailing day and see perfectly capable (and usually expensive) boats motoring upwind instead of sailing. (These are the same boats you see running dead downwind with the boom trimmed to the centerline of the boat and the roller-furling jib on the wrong side.) To these sailors, beating upwind is too uncomfortable, difficult, frustrating and slow. Beating upwind requires skill, perseverence, heeling, spray, pounding, trimming in the headsail, scary noises, and attendant discomfort. It is difficult to hold your cocktail upright and maintain a calm demeanor for your guests with your ascot perfectly tied while your boat is pounding into the waves at a 30 degree angle. Plus, your wife may not sleep with you for at least a week if you persist. Nevertheless, on a given desirable cruising course heading with wind from no predominant direction, if you care to sail, you will end up going to weather at least 1/4 of the time (= 90/360 degrees) and downwind 1/4 of the time. So, you will effectively spend half your time beating or running downwind.
Running downwind can be equally scary. Some sailors have balls and some don't. Those without should probably just give up, spare us all the embarrassment, and buy a trawler.
Almost all boats sail well while reaching. The greatest speed differences are achieved upwind or downwind.
I am a bit mystified by those who don't like to go to wind. It is definitely my favorite point of sail. My boat, like other S&S boats of its era, was designed to go to wind and when she settles into a groove, it is thrilling. But I can recall it wasn't quite as much fun when I had a much more tender boat that didn't point as high. And as to the original post, yes, pointing ability makes a big difference, racing or otherwise.