I don't think it is a good comparison, as bad as comparing a bus with a Porsche on the interstate highway
You could compare on the interstate a small family car with a big limo but that is not fair also because the limo is not more difficult to drive or handle, at least on the interstate, while a bigger boat can (or not, depending of size and type of boat).
I believe that for most regarding big boats the main issues and questions are:
How big can I sail solo or with my wife comfortably?
How will the size of a big boat affect maintenance and marina costs?
How that size will affect the ability to put in and out the boat from marinas and ports alone or just with a bit of help from my wive?
How much that big size will affect availability of marina and port spaces on the cruising grounds I will sail?
How much I will gain in speed regarding a less expensive to maintain and easier to maneuver (at the marina) boat?
How much space and carrying load do I really need for the type of sailing and life-style (use of the boat) I have or want?
Sure there are many factors to consider. And the monetary considerations are with us in almost anything we do in life.... buying a car, a home, a boat, etc.
Of course the main use of the boat is the primary question.
If it will be used as a weekend coastal cruiser, that is quite different from a liveaboard / blue water / long range vessel.
That is what my wife and I do with our boat, thus the maxim: buy as big as you can afford.
If we could afford it, we would have a Perini Navi faster than you can say 'sold'.
But, just to throw a debating point into the mix, I have found that a smaller boat is less forgiving, i.e., they react much quicker to wind changes than a larger vessel. So, if I am below making something to eat, or whatever, and the wind goes from 15 to 30 + knots, our CT 56 handles that even while flying all the sheets.
IF that happened onboard a smaller sloop, it would be a much more urgent situation to handle ASAP, with potential disaster hanging in the balance.
So, there are more positive reasons to have a bigger boat, than a smaller one, once your wallet is a lower priority than the ocean is.