1. More halyard tension - there is no more ME to tension the halyard with. Winch on mast is a #30, 2 speed. I still have a foot or so of halyard at the top of the mast, luff is real tight.
2. More outhaul tension. I may have something to work with there. Just checked, I had about 10 inches of adjustment left. The adjustment, with sail down, was made very easily, very little tension.
3. Vang was bar tight when I took this photo.
4. Iíve no traveller. Just a lad eye on the aft coach roof (center cockpit).
It helps to talk things out.
1) With regards to the halyard tension, you should be able to get more a couple ways. Make sure that the boat is up in the wind with the vang and mainsheet eased and then really honk on it. You should be able to get more tension but maybe not since the stretch in the halyard may be all that is causing the appearance of an ease. Plan 'B' might be to go old school put a handy billy (small tackle) on the halyard with a rolling hitch and then use a winch to trim the handy billy.
3) To increase vang tension, once the sail is fully up, and while the boat is head to wind really crank down on the mainsheet, that has more mechanical advantage than the vang. Then tension the vang as hard as you can. That should result in greater vang tension. Few cruising boat vangs can be tensioned under load without being led to winch.
Can you rig a Cunningham? Would do the same as increasing halyard tension. Or sail with a reef in and more tension on the tack line.
In the absence of a traveler can you rig two mainsheets toward the back(aft) part of the sail. One anchored at each side of the boat or the house. You have the hardware for one. Just need itís twin and two new anchor points. This would serve the purpose of a traveler going upwind and vang only needed downwind. Or move one out to the rail. This would give more vertical downward force where you need it. On a prior boat used the tackle designated for use as a preventer temporarily directly below the boom end to bring it straight down went going upwind. Mainsail was stretched out and this helped some.
A cunningham is the opposite of what is needed. A Cunningham does not do the same thing as tightening the halyard. The Cunningham only tensions the luff of the sail. Because of that, when a Cunningham is used, the leech is relatively loose compared to the luff. The main halyard tightens both the leech and the luff at the same time. Since there are small scallops in the luff, I figured that both could use more tension.
Once the main halyard is tensioned, the handy-billy can be taken to the leeward rail and used to tension the leech by pulling against the mainsheet to create a downward vector.
You mentioned twice that the leach flutters excessively but no one has suggested tightening the leach line.
The sail has a pretty nice flying shape. Tightening the leech line will cup the trailing edge of the sail between the battens- increasing heel and weather helm. That said, its an older sail and if all else fails, then a small amount of leech line might help.