Quote:

In a real dynamic situation however the mass of the mast had a tendency to keep the boat from capsizing due to the conservation of angular momentum

No doubt you could calculate this using momentum, energy or inertia calculations. I find inertia calculations to be more intuitive.

If I look at my boat, I see the predominant inertias (around waterline) as:

Keel at 8200 lbs (3727 kg) , at about 1.5 m gives 8400 kgm^2 inertia

Mast 60 feet, 550 lb, 250 kg, at 10 m gives 25000 kgm^2 inertia

Fiberglass hull 11,000 lb or 5000 kg at 1 meter gives 5000 kgm^2

First I notice that the mast and rigging determine the predominant inertia, so yes the mast is the most important inertia that will resist acceleration. So if we have a fixed torque applied to this inertia, it will cause an angular acceleration much higher if the mast is missing.

The sum of inertias is 38400 kgm^2

Now lets put a 100 lb mass or 45 kg at the top of the 18 m mast. This gives an inertia of 45 * 18^2 = 14580 kgm^2

This added inertia is about 38%. So yes, this added inertia will also reduce the acceleration.

Just for kicks:

The derivative of my stability diagram is about 100,000 Nm/rad at zero. This is the effective spring rate of the water against the boat.

So the natual frequency of my boat is:

Sqrt(100000/38400) rad/s = 1.61 rad/s = 0.257 hz

so the natural period of my boat is about 4 seconds.

Without the mast, the natural frequency is:

Sqrt(100000/13400) = 2.73 rad/s = 0.434 hz or 2.3 second period

So if we just consider inertias, the angular acceleration will higher when the mast is missing. On the other hand, the angular displacement is the same with or without the mast because the frequency has changed.

However, the problem comes that we have neglected damping. When you reduce the inertia, the acceleration is higher, the velocity is higher, so the damping is higher. Since damping is proportional to angular velocity, the damping is going to be higher by about the ratios of natural frequencies: 0.434/0.257 = 1.7 times higher without the mast.

When we consider the higher damping and the higher frequency without the mast we find that the heeling angle goes down without the mast because the damping is higher.

Here is a simulink model that illustrates the effects.