You should have an on / off switch for the house bank that turns off the entire panel. You should also have a 1 / 2 / all / off switch that allows you to start the engine with the cranking battery, or the house bank (in case you have trouble with your cranking battery). That's the setup I prefer anyway.
The bilge pump should be wired directly to the battery (with an inline fuse). Some people also like to wire the VHF this way too.