I had them on both last night in order to recharge them.
You mean while you were on shore power? What kind of shore power charger do you have? Mine is a Xantrex TrueCharge 10TB. It has two separate "channels" - one for each battery. Its output connects directly to each battery. The battery switch need not be on for the charger to work.
The only time the battery switch setting should matter for charging the batteries is when you're charging from the engine's alternator.