That sounds like a tuning issue. Have you sighted up the mast while you are sailing on each tack? The mast may be straight dockside, but when you put it under load you may find that the stays are not at the same tension. Be careful, it is possible to overtune which will greatly reduce the lifespan of your standing rigging.
I watched the Brian Toss video on rig tuning and found it pretty informative. If it's within your budget I'd recommend a professional tune with someone who is willing to teach you about what they are doing.