I'm thinking your sail is the problem. My last boat had that situation as well. The boom hung way down and looked all wrong. It sailed fine, but jibing was a little more excitement then we needed.
Would a blown out sail do that?
1968 Morgan 24/25 S/V My Fair Leslie