Unless I'm confused the OP said he has a list to port (more weight on port side) and he consistently sails faster on a starboard tack. If weight distribution was the cause you would think that the boat would be faster on port tack because it would sail flatter (as the OP said). I would suspect rigging tension but there are a lot of unknowns to explore.
SV Laurie Anne
1988 Brewer 40 Pilothouse