Vibration analysis is a very complex science; changing the mass, changing the geometry .... and changing the forces acting internal to the system will change the 'natural frequency of oscilation' of the vibrating components.
Simple speak: If by tightening up the stuffing box radically alters vibrational performance would seem to positively indicate that the system is 'out of balance' and the tightening is merely 'shifting' the vibrational characteristics to 'another frequency / amplitude'.
It would very strongly suggest that 1. the coupler and engine alignment are 'out of whack' 2. the propshaft is bent.
1. Realign the engine to the coupler, aiming for 'perfect' alignment ... ie.: less than 0,001" of run out as measured between face to face on the mating surfaces of the flanges. Of course you will on an old boat probably will find broken or 'frozen' motor mounts.
2. With the stuffing box 'released' (or packing removed or loosened so that the packing is no longer in contact with the shafting) use a dial indicating micrometer positioned near midshaft of the propshaft, rotate the shaft and measure if the propshaft is 'true' (virtually no deflection of the dial indicator) as the propshaft is turned through 360 deg.), Or remove the propshaft, take to a machinist (with a large 'set-up' table --- a huge heavy steel table calibrated for its flat surfaces) to verify the 'straightness' of the shaft, etc. If this is a shaft that has been in the boat for years of service, its 'much better' to remove it, have it re-'trued' by a machinist ... and have the 'journal area' where the stuffing box packing and the cutless bearing 'dressed' to remove and developed surface roughness and pitting of the shafting.
For a DIY validation/correction --- do websearch or forum-search on the proper methods for engine re-alignment and stuffing box repackiing methods ... and realign and repack with the goal of 'perfect' alignment and uniform 'side loads' on the shafting from the packing, etc.