Standing Rigging Question
This very well could be your imagination since you have no objective means of measuring or it may be a whole range of issues. You mast may not be plumb. The shrouds may not be evenlt tensioned. Your knotmeter may not be on the centerline of the boat or may be cocked to one side of the boat. Few production boats really are perfectly symetrical. Your mast step may not be centered. Measuring from your stem, check that your shrouds and jib sheet leads are in the same place on the deck. Your outboard may drag on one tack and not on the other. Your keel or rudder may not be symetrical and so on.
There are things that can cause the appearance of assymetry. Current can really change the appearance of how well a boat can point on one tack or another.