I would like to install both a hi-gain VHF antenna and a hi-gain WiFi antenna on my masthead.
That'll work for the VHF (with caveats), but not for the WiFi.
1: Is there a significant interference/shadowing problem doing this?
Unlikely to be appreciable RF interference between the two. 802.11g would be at over the 14th harmonic of the marine band. 802.11a would be at over the 22nd and 30th harmonics.
Radiation pattern interference: Probably. Probably not an issue for the WiFi antenna, as there are much bigger issues, there.
2: Is there a separation distance I need? How do you get it?
Brackets. Long brackets.
I've been out of the craft for years and years, so I'm a bit rusty, but, if
I recall correctly: You want at least
1/2 wavelength separation between the two elements at the longest wavelength (lowest frequency). That would be on the order of 1m for the marine radio antenna.
3: Is there a better way?
Yes. Forget putting a WiFi antenna atop the mast.
4: Will the cable loss up to the masthead negate all of the gain in the antennas?
For the VHF: Depends. If you use Belden 8214 or equivalent, your loss, by the time you add connectorization, will probably be about 2dB. Height is king on VHF, so that's probably acceptable. Even if you want to avoid the weight aloft and go with RG8x, it would probably still be a net win. I you go RG8x: Offset the increased loss, some, by using BNC connectors where you can, rather than the so-called "UHF" style commonly used.
But the WiFi... The WiFi is a whole 'nother ball of wax. That's 3.7 and 5GHz for 802.11a, 2.4GHz for 802.11b/g, and 2.4 and 5GHz for 802.11n. Even with Belden 9913 (expensive and a PITA to work with), loss would be on the order of 9dB or more at 2.4GHz once you added connectorization, and in excess of 12dB at 5GHz. Belden 7810 wouldn't be a whole lot better.
And don't forget: That loss works both ways: Both transmitted and received signals.
All loss figures based on 100' of coax.
5: Will the tighter lobes of the high gain antennas give me a false gain when one factors in the mast movement?
Gain has gotta come from somewhere. For omni-directional antennas it comes from flattening the radiation pattern on the vertical axis. Thus, when you heel, the windward radiation heads for space and the leeward radiation heads right into the water.
For your WiFi: Put a high-gain antenna up 40 feet and, never minding the loss figures quoted above, all your signal will be headed for the horizon... 40' up. Sitting right under the antenna, down on the deck, you might see nothing. (I'm just guessing, never having tried this.)