meters and troubleshooting...
DC ammeters that are rated to carry more than about thirty amps have to have a 'shunt' resistor. Given the fact that all of the current flowing to or from the battery needs to be measured, this shunt resistor is usually very close to the battery physically.
Take a look at your battery's hot leads. They're probably coming from a 1/2/both switch. Follow the common lead from the switch and see if it leads to some kind of metallic looking block with the large battery cable from the switch attached at one end, and another large battery cable wandering off, probably toward your circuit breaker panel, starter motor, whatever. The big metallic block (especially if it looks like a heat sink with fins, etc,) is your meter shunt.
What it actually is can be described as a very small value, very high current resistor. Normally it will be less than one ohm if you have a meter. On either end of the shunt resistor, you'll probably find a small wire, and these go off to the meter. What they are doing is reading a very small 'voltage drop' across this resistor. This, in turn, can be easily read directly by a meter, and the meter is calibrated to read in amps, rather than voltage.
In the case of boats, corrosion is your enemy. Bouncing meter needles is usually an indication of bad connections somewhere. I'd start with the batteries. Remove and use an old fashioned battery terminal wire brush to clean the posts and the battery connectors, then reconnect everything, making sure it's good and tight. Finally, put the anti-corrosion stuff of your choice on the terminals. (I like Corrosion Block).
After checking the battery ends, move to the other ends of the heavy cables. Make sure that there is no rust, slimy green stuff or whatever, growing on the main ground, which should be on your engine block. In all likelihood, there will be a bunch of wires there leading off to your bonding system, electronics (radios like REALLY good grounds!), etc. Clean all of these connectors up, reinstall and tight them, and slather them up with anti-corrosion stuff.
Next stop is the alternator. Make sure everything is snug. Look for connectors that have melted or even gotten really warm. Factory wiring is almost always undersized, or at best, the minimum they can get away with. Take a hard look at the output of the alternator and make sure the wire is up to the task of charging a nearly-dead battery at whatever current the alternator can put out. Bigger is always better with wire, no matter what the yard or factory people tell you.
Check the leads going to the main DC circuit breaker and make sure they're tight. Moving the wiring harness to look at stuff, vibration, and our old friend corrosion all conspire to make those leads a high-resistance source.
Finally, you can tighten up the leads on the meter itself, plus wherever they seem to come from.
There are ways to test the meter, but in most cases, there's no way of knowing exactly how much voltage it takes to get full deflection on the needle. Sometimes it's as simple as connecting a AA battery across it. Other times that would blow the meter up. Without the right test equipment, it's easy to blow something up.
Last, but not least, if you do happen to have a good digital meter, you can disconnect the common lead from the battery switch and put the meter into the 'AMPS or MA' load. In this case, the meter goes into the circuit IN SERIES with the cable. Make sure you have a fairly small load, like a VHF or something, and nothing else turned on and see if the digital volt meter's reading agrees with the boat's built-in meter. (Plus or minus about ten percent, anyway.) Remember that your DVM will have a limited current capacity--usually 3 amps, so don't try to fire the engine up, or something like that. Just a small load will do the job.
If the DVM and the boat's meter agree, but after putting everything back like it was, the meter still pegs, you've got a problem. If you have a built-in battery charger that works from shore power, test it. I had two of them go bad. One of them went berserk and hit my brand new Rolls batteries with 51 volts and nearly twenty-five amps of AC current when one of the main bridge rectifier diodes went out. The other one was harder to find: it never dropped from it's mid-level charge of 13.6 volts, and slowly cooked the water out of the batteries.
Which brings me to one last comment: if you have an old battery charger, bite the bullet and replace it. The old Consta-Volt systems that a lot of older boats have are simply junk. If you're using an automotive charger that is over ten years old, relegate it to the garage and get a new charger. The same is true of very old voltage regulators. If it's over ten years old, you might consider getting a new one with at least two-stage charging abilities built into it. It'll keep you from frying your batteries. Spending the bucks on a new charger will pay in the long run, because you won't be boiling the batteries constantly.