Well.. sailing under main alone always restricts windward ability. So i wouldn't form any opinions based on that day's events.
Here's what I'm sort of getting from this... sail more, think less. I know, I know.. that's horrible advice. But I rather get the feeling that you are a well-read, analytical guy. And also, that you haven't had that much time at the helm of this boat yet. Sailing is a mental game, for sure. But sailing also has a lot of touch stuff... and that just takes time.
I think, my friend, you are just thinking too hard. Yes, the mast rake may need to be adjusted. Maybe the main sail is blown out, too old, too small, too battened, not battened enough, maybe the halyards aren't tight enough, maybe the cunningham is too tight, maybe the boom rides too high or the backstay is too loose or the vang is too far foward on the boom to be effective or the mainsheet isn't far enough aft. Could be that the headstay is too loose and that lets the masthead fall off in a puff, or it could be that your jib sheeting angles are too tight so the jib foot is too full. Might be the leach line on the main is too tight, and that's building too much belly in the sail and causing the angle of attack to be too fat. maybe the lower gudgeon is out of plane and the rudder is entering the water at a bad angle and causing laminar deflection. Maybe the centerboard trunk is restricting the centerboard in some way, and maybe martians are using your mast as a remote antenna for their plankton mining operations.
But, here's the thing.. there's no way to know. Those are all relative issues, which is to say, there's no 'correct' leach line tension. You just either loosen it or tighten it. Sure, there are rules-of-thumb, but at the the end of the day it's a judgement call. It's hard to make that judgement until you have a good idea what 'normal' feels like.
And, to wind up this book, I think maybe it's time to just sail for a few trips. Relax, see what the boat feels like. She sounds like a safe, basically well-found boat. You beat her up a bit, she held together fine. So now it's more about fine-tuning things.
At least, that's what I'd do. Just sail the 'ol gal for a while.. few days, week, month, whatever. Then you won't wonder if the sail is too small or big.. you'll be able to look at the boom and masthead and know if there is too much/little room there for more canvas.
... or I'm wrong.
Living aboard, currently in the Chesapeake
O'Day 37, still new to us