[Sailormon6]

Sailboats work their best when the sails are balanced. When I’m trying to decide whether to reduce sail area on a boat, I ease the less powerful sail [the jib on a fractional rig, or the mainsail on a masthead rig] while beating to windward, until the sail is completely luffing. If the boat is overpowered and heeling excessively in the puffs while sailing on the most powerful sail alone, then I know it is time to reduce the area of the most powerful sail. When I reduce the area of one sail, I also reduce the area of the other sail proportionately, to keep the sailplan balanced.

If you have reefed the mainsail as far as you can, and are flying your smallest headsail, and the boat is still overpowered, then you have stayed out too long. In that case, if my headsail was hanked on, I would probably lower the headsail and use the fully reefed mainsail alone to get back to shelter. If my headsail was roller furling, I would lower the mainsail, and roll out a very small headsail to get to shelter. You should avoid being in that situation in the first place, however, because any sloop (fractional or otherwise) that is sailing on only one sail (either the jib or the main) is hard to maneuver, and doesn’t always go where you want it to go (as you have seen).

You said you were unable to get up enough speed to tack the boat across the wind, because you were feathering to windward, and the boat heeled excessively when you bore off to leeward to gain speed. That means the boat was grossly overpowered. If you bear off to gain speed, the boat might heel excessively and capsize. If you don''t bear off and gain speed, you can''t tack the boat across the wind. There is no solution to that dilemma. The boat is not designed and rigged to stand up to that much wind.