As you have seen, your question is pretty much the kind of "how long is a piece of string?" question which is bound to elicit a a diversity of seemingly contradictory answers. The mutually exclusive seeming responses above reflects the fact that there is no one simple answer that applies to each and everyone who might ask this question. The one right answer for has to start with your own tastes, goals, venue, needs, sailing and physical capability.
There is bound to be disagreement on Even how to measure an answer of 'how big?'. To me the best measure of the ease of handling, seaworthiness, motion comfort, carrying capacity, starts with a boat's displacement. Historically, single-handed cruisers typically fell within a displacement range of 5,500 to 12,000 lbs. With modern sail-handling and boat handling hardware, that displacement range has grown tremendously. But that expanded displacement range comes with a price tag; a price of greater complexity, maintenance, and cost.
And at whatever displacement meets your needs, within reason, the boat with the longest waterline at that displacement will generally offer the best balance of performance, ease of handling, seaworthiness, carrying capacity, and motion comfort. In my experience the easiest single-handers have a design L/D less than 200, with a sweetspot somewhere a around 160 or so.
Then comes rig choices. While traditionalists lean towards cutter and ketches, if preformance and ease of handling is important, than a fractional rigged sloop, with its smaller headsails and ease of powering up and down, makes an ideal rig. In my mind, the ideal single-handed has a SA/D in excess of 20 or so with 22 being a sweet spot due to the ability to get by across a wider wind range with fewer sail changes and non-overlapping jibs.
Stability is also of key importance to a single-hander since it allows the boat to carry its sail area through a wider wind range. As is an easily driven hull form. The combination of an easily driven hull and lots of stability help at both ends of the wind range.
For me, these numbers resulted in an 10,600 lb., 38 foot boat with a L/D of 165, and a SA/D around 23, a 38% ballast ratio, 6'-4" draft which I find to be a very easy boat to single-hand.
Then comes sail handling gear. This is also a very controversial topic. After single-handing for over 50 years in a wide range of conditions, I am a firm believer in having all if the control lines (including halyards and reeling gear) within easy reach of the helm. But to put that opinion in perspective, Jon Eisberg and SVAudacious, who are probably two of the most accomplished single-handers on this forum, prefer the halyards and much of the control lines at the mast. The point being that there is not a one size fits all answer on this.