Presumably the folks on the high side are as far or farther from the axis of rotation than the folks on the low side. So their righting arm is as long or longer.
I say "as far or farther" because the center of buoyancy shifts to leeward, which I think implies that the axis of rotation shifts to leeward as well.
Your first paragraph presumes wrongly that righting arm is only related to distance from axis of rotation. There's a vector component too. Righting moment is proportional to the horizontal component of the distance, as it's gravity acting on the weight, doing the work.
Think of 3 people on each side, with the boat at an extreme angle of heel. The 3 on the high side could be directly over the centre of rotation and despite being a long distance away from it, the horizontal component of their distance is zero and hence they provide no righting moment.
The 3 on the low side are the same distance from the centre of rotation but their horizontal component is maximised. They are making the boat heel more.
Generally the effect of ALL weight above the waterline is to reduce the righting moment, and the more the boat is heeled the more the effect.
The exception is when the weight is on one side, but the effect of that still drops as you heel and can reverse in effect at extreme angles.