I don't know your rig. It sounds like your vang was causing the mainsail to be too powerful.
Generally lots of tell tails to indicate how and where the air is travelling over the sail are a good idea.
The jib determines the course you can maintain so get the forward tell tails drafting nicely then work back. On the mainsail, the top 1/3 of the sail does the most work so always trim the sails starting with the top tell tails then work down.
Also start with things loose and gradually tighten the adjustments to figure out what is affecting air flow over different parts of the sail.
When in doubt, let things out. Selectively of course, but the natural instinct to tighten things is not always productive, tighten something, look at the sails then loosen and check that what happened was logical.
Of course when the wind is increasing you need to tighten thing but its also easy to overtighten so loosening off to check trim is a good check while you are figuring out what works for your rig.
Do this long enough and you will find you get the tell tails on the mainsail and headsail moving in tandem as the flows change over the set sails.
Don't think about power, think about efficiency, keeping the air flowing all over the sail and keeping the forces as low as possible in the sails and the wetted surfaces.
Its nice to have everything set so that when a gust comes through you can take the lift heading up a bit then resume coarse when the gust has gone. Its easy on the crew and easy on the rig.