Guage is really n/a when it comes to coax. What you use depends on the Ohms (typically either 50 or 75) and the distance you need to run it. Refer to the recommendations of the radio and antenna.
is correct that guage is not applicable when talking about coax and it comes in 50 and 75 Ohms Impendance. 50 Ohm Impedence is for VHF service
, while 75 Ohm impedance is for television antennas and cable service.
In general the thicker the cable insulation the less loss there is to the signal.
There are five types of coaxial cable found on boats:
RG-58/U is thin (3/16" OD) and used for short runs to connect electronics etc
RG-59/U (75 Ohms Impedance, 1/4"OD) Used for Cable and TV antenna work.
RG-8X (50 Ohm Impedance, 1/4" OD) is common for VHF and HF work up to a point where signal loss (attenuation) is too great because of cable length
RG-8/U ( 52 Ohm Impedence) and RG-213 (50 Ohm Impedence) are used to get max power to the antenna for both VHF and HF service. Both are about 13/32" in OD
This does not cover all of the cables but should give you a good start and check to see what your radio and antenna require and also what the cable manufacturer says as well. You will also have to pay attention to how much bend radius you can put in the cable.
You will also want to use marine grade connectors matched to the type of cable you are using.
Pay attention to your materials and equipement and you will have an install which will give you the best performance.