When you tack a sailboat, the sails stop driving from the time they start to luff until the wind fills them on the opposite tack. That means the boat has to coast through the turn. If it doesn't have enough momentum to carry it all the way through the tack, it'll come up to the wind, and then fall back on the same tack.
When you try to tack while beating to windward, if a big wave slaps the bow, it can stop your forward momentum. Also, the wind on the hull slows your momentum. Also, a heavy boat will carry farther than a light boat. These factors are not a problem in moderate winds, but as the wave size and wind speed increase, they limit the boat's ability to coast far enough to get all the way through a tack.
Having the right sails for the conditions is a good start, but it will also help if you make sure you are making as much speed as possible before you start to tack. You need to put the helm over more aggressively than normally, so you get the bow across the wind before the boat loses it's momentum. When you start the tack, watch the jib
, and don't release the working jibsheet until the jib
is completely luffing, because, even after it starts to luff, it'll continue to provide some drive, and you need every little bit of drive you can get, to carry the boat through the tack. If the prop will stay in the water, start the engine, and you can use it to help the boat get across the wind, or you can just take the sails down and motor to shelter.
There's nothing wrong with jibing in those conditions. I've done it on my 35' boat in stormy conditions. If you're sailing toward a lee shore and you need to tack for sea room, you need to change course somehow, and you have to do what works.
But, the best practice is to avoid those conditions. An outboard powered coastal cruiser is designed for sailing in moderate conditions. When the conditions are so challenging that you're having trouble tacking, you're past the time when you should have found shelter. Coastal cruisers are characteristically light in weight, and big on windage, and they're inherently difficult to handle in big winds and seas. Sometimes you get caught out, but if that happens, find shelter as soon as possible.