There should be a position where no lights are on, if not you have to shut off power somehow to keep from draining the battery. The center position is the off and on the terminal side should be the where power comes into the switch. Then flipping the switch one way should turn on a set of lights, flip it the other way should turn on a set of lights as you said. What concerns me is when you flip on just the bow and stern lights and they got dimmer, that sounds like the lights in that position are wired in series. My schematic, and not all boats are the same but wiring logic is and all lights on a circuit should be on a parallel path.