The fact that even the mast shakes violently, and the fact that the problem occasionally disappears really points to the prop not opening. But it would be nice to know what sort of prop it is before committing to that theory.
".. there is much you could do at sea with common sense.. and very little you could do without it.."
Capt G E Ericson (from "The Cruel Sea" by Nicholas Monsarrat)
1984 Fast/Nicholson 345 "FastForward"