I've had a busy weekend
Phew.. this one's gonna be a toughy, mostly because of memory.
Making the decision to sail depends on decisions by the captain and mate, as well as the reasons for sailing, and takes into account the type of vessel.
One of the biggest seas in memory was several years ago en route from British Columbia to San Francisco on a 151' steel brigantine. If I recall correctly, they were about 15' feet rollers with some breakers, and 40-50 knots of wind. I think the weather lasted a little over a day, and our boat loved it. If I recall correctly, we we set our fore and main staysails, and our mainsail was undersized, so we reefed it and set that. Honestly, I felt really safe on that boat during this weather. She sailed well with a following sea, and seemed to just ride over most of the waves.
My first ship like I mentioned earlier was a 112' wooden brig, and way more like a washing machine than anything else.. I think that time she was making 11 knots was in 30 knots of wind, and waves high enough to break over the chest high caprail.. so perhaps 10 feet?
Another boat I sailed on was a 106' steel ketch with squares on her mainmast and a very shallow draft of 3'.. she was AWFUL against any head seas or winds, and felt like riding an ATV in any kind of chop. I remember a trip heading north from Morro Bay to San Fran, and all but myself, the cook, and the captain seasick.
The most squall-y sailing has been in the South Pacific with a 140' steel gaff-rigged ketch (converted from a 1950s fishing boat). I don't remember the windspeed of it, but a squall we encountered not only shredded our staysail (which was the only thing we had set), but bent our 5" diameter steel bowsprit to one side. She was great for the most part to sail in, but when she was fully loaded (used as a cargo sailing boat) to her plimsoll line, she was very roll-y and sluggish.
Other ships less than 100' have been schooners, and most of those have been dependent on passengers, so the weather would certainly be taken into consideration. As mate on a 60' steel schooner, I would often make the call with the captain whether or not we would take passengers out. Though the schooner has been in much heavier weather, usually we would decide against sailing in anything higher than 20 knots, simply because it's not a comfortable (or dry) ride for passengers. On the other hand, with just crew we raced down the Chesapeake Bay on a 104' wood schooner in 25-35 knot gusty winds with both headsails, foresail, and a reefed mainsail, and had a great, if slightly nervous, trip in less than 18 hours.
As for stuff I don't like to sail in, it really depends on the boat. The windage of Bounty, due to her higher bulwarks and freeboard, would respond much stronger and list more heavily than most schooners of similar "Class A" size (if no sails are set, of course). Also, the shape of hulls will also determine how a boat fairs in conditions. There's no real answer other than being familiar with a boat, being aware of any "issues" with it, and trusting that vessel will get you home.