I might try using a couple of instruments to measure and compare speed. Check your speed over ground with a GPS, get a knot stick..to compare to your paddle.
There's also the effect of 1 to 3 meter waves...approx..3 to 6 ft..in americanese..
Is it always the same angle of attack to the waves?
I know you don't have currents as we know them..here near the ocean...but I believe you have some current up there due to river flows, wind driven, outflows from storm sewers etc....
Waves, and currents will affect speed. Here on the hudson river...we have a big underwater profile called a keel...if sailing at a right angle to the current..it will have a greater effect..
The fact that your speed is affected always on the same tack..may rule these issues out...but something to consider.....