My instincts tell me that Pigs is on the right track. Any design or installation issues should have been sorted out much earlier.
One analysis is that either the cooling system, assuming there is one, has a problem, or something internal, such as a bearing, is failing, causing excessive heat generation.
It's possible that the fluid cooler is cracked and is leaking. I would look at it first. Flush it out and pressure test. There might be a hose or line
leaking if the cooler is external to the transmission. This analysis is saying that the fluid escaped and then the trans overheated, not vice versa.
It's possible that if you were trying to push the boat past its hull speed, the trans was simply overloaded, with slippage causing the heating.
You will want to flush out all of the fluid to eliminate the residual burned fluid remaining from the overheating. Removing and flushing the cooler will help achieve that end. If, in the flushing process, any solid material is dislodged, the cooler should be replaced (discard the old one; it's NOT suitable for a spare) and consideration be given to inspecting the transmission.
Just my thoughts.