The fact that even the mast shakes violently, and the fact that the problem occasionally disappears really points to the prop not opening. But it would be nice to know what sort of prop it is before committing to that theory.
1984 Fast/Nicholson 345 "FastForward"
".. there is much you could do at sea with common sense.. and very little you could do without it.."
Capt G E Ericson (from "The Cruel Sea" by Nicholas Monsarrat)