OK, I haven't really thought this through but... maybe you actually lack momentum. That same boat is only moving at 1 knot. The faster you go, the more stability you have. Look at a bicycle. Hard to balance when stopped, the faster you go, the more stable you are. The better you can withstand side forces.
Several times when motoring or sailing into a strong current I noticed that the helm became more tender than normal.
At first blush it would seem that it wouldn't matter if the boat was moving through the water or the water was moving past the boat.
My experience is that as soon as the current speed gets close to the boat speed through the water the helm gets very tender.
I think I have figured out why.
When motoring directly into a 4 knot current at 5 knots you get an over ground 1 knot progress. If however due to wave action or even slight steering adjustments or even current eddy's you get even slightly off dead ahead the vectors change rapidly and the bow gets pushed off affecting the helm.
Have you experienced the same thing?
Does my explanation make sense?