Battery chargers need to see a load so it's almost a certainty they're wired directly to the batteries and thus independent of the switch.
You should look for a smaller gauge wire attached to the battery and trace that back to your charger.
You didn't ask but the same isn't true for the alternator charger. Usually that's the same heavy gauge wire that goes to the starter from the common output on the battery switch. So when you're under engine power, be sure to switch to both/all batteries.