CloseCall - here's a fairly straightforward method to analyze if your sail is at the correct 'shape' for your boat and for your present sailing 'expertise'.
The following article I wrote many years ago was written in mind for dacron sails made by a now generally obsolete method at a time much before very 'stable' dacron sail cloth was available. Woven dacron sails made since ~2000 are constructed quite differently and dont need to be 'raised' with an additional 'stretch-out' luff as was needed
for dacron mainsails made before ~2000. Hopefully the old article will help you evaluate if indeed you need new sails or just some minor 'adjustments' by your local sailmaker to restore the proper shape when 'raised' ... or you 'just' need to add some ADDITIONAL halyard tension 'after' you raise that sail. https://forums.sailboatowners.com/in...insail.120970/
With old baggy, thread worn and dirty sails, one can easily assume that your mainsail is of the 'older' design criteria'.
With the methods outlined in the above link ... your goal will be to so raise the sail such that after doing the proper additional tensioning after raising, the corner of the sail thats at the intersection of the mast and the top of the boom (the tack) is at near 89° (90° is good enough). You can simply use a moderate sized piece of precut and 'square' cardboard to visualize how close to that 90°. Once youre sure the 'stretch-out' is proper (first evaluation is done at-the-dock on a windless day), then go sailing in moderate winds and note the angle that aft end
of the #2 batten makes with centerline of the boat when closehauled/beating ... the aft end of that #2 batten should be closely parallel to the centerline ... adjust/fiddle with ALL sail tension controls (including varying tenson on the mainSHEET) until that #2 batten is parallel to the centerline ---- when beating/closehauled sailing. If this 'old' sail can't come close the that tack angle requirement (by proper 'stretching that luff' and you can't get that aft end of the #2 batten end - 'straight back' & not
'hooking up' towards windward), your boat *will* be SLOW, will have a quite noticeable 'weather helm' and will tend to aggressively heel over even in 'moderate' winds. With this 'evaluation' method you can make a fairly close assessment and determination that you need to have a relatively inexpensive 'boltrope' adjustment by a sailmaker, a slightly more expensive boltrope replacement, or you really need 'new' sails.
BTW - make sure that your backstay (forestay) has proper tension before you do the sail evaluation on the water, as a too loose forestay will cause similar problems as you describe.
Further, when sailing close hauled and with your hands FREE of the tiller/wheel and the boat should be able be tested be able to continue sailing in a straight line
... no luffing up, no heading downwind when your hands are free of the steering. The adjustment to do this is by tensioning the mainsail HALYARD up/down until you can do so; but then, followed by a slight easing of the halyard tension until the boat, when beating, starts to 'slowly' head up with your hands free from the steering ... then, make a mark on the halyard and the mast, so you dont have to often repeat this process for when sailing in normal/moderate winds and waves.
If you can't obtain this 'dead fish' neutral helm, then 1. consider to have that bolt rope 'eased' or, 2. replaced, or 3. new sails.
Usually with a still decent quality but 'old' mainsail that can't be 'shaped' on the fly due to a shrunken boltrop, 90% of the time a re-set/re-adjusted luff bolt rope will restore the proper & reasonable sail shape, reduce the aggressiveness and amplitude of the heeling, bring the boat back to 'proper' speed and with damn little adverse weather helm.
The newer dacron mainsails dont need this 'stretch-out', are already cut and designed to be simply raised and forgotten with respect to luff/halyard tension. The old design sail unless you 'shape' them by additional tension on the halyard/luff, 'after' raising, will have a very 'baggy' appearance, the amount of draft will be 'god-awful', the point at where maximum draft occurs will be aft towards the leech, and that aft end of the #2 batten will be quite radically pointed up (hooked up) to the weather side of the boat.
Yeah, this a lot of stuff to go through ... or you can just plunk down several hundred $$$ for new sails.
If this confusing (it well may be) or if you need addition question answered .... dont be bashful. My bet is that you aren't raising this OLD sail properly, and that's causing 90% of the problems that you're experiencing. Improper sail 'shaping' (after raising) is a very common problem, especially for 'self-learned' sailors.
hope this helps. ;-)