Think of a seesaw. The center is the point of rotation - the boat equivalent is the centre of lateral resistance, the point the boat rotates around. Put only one person on the seesaw, that becomes the centre of effort the point where the force can be considered to act. With only a jib that point is ahead of the CLR so the boat turns that way, because it is not offset by a counter rotating force on the main.
The swing keel makes some difference to the position of the CLR ie the further back or raised it is the further back the CLR so having it raised would add to the imbalance of the sails.
Downwind the centre of effort is offset to one side so this tends to produce a rotation akin to pushing on one quarter. However that may be readily offset by the rudder in light conditions.
The general point is when reducing sail to balance the sail plan by reducing jib and main - an option which was not available to you. The slugs should not pull out of the track. Perhaps there is excessive wear or a mismatching of sizes.