+1. We've got a full keel with a barn door rudder, and I'm here to tell you backing is ALWAYS an adventure. If there is absolutely no wind and no current, I can usually know how the beast will track in reverse. Mostly it's all about using the throttle intermittently to balance prop walk with having the rudder pretty hard to starboard (the rudder imparts very little turning effort when in reverse.)
However, when conditions are not totally benign the old gal loves to make my life interesting. Doing the mental gymnastics thinking about the opposing forces keeps me on my toes, but usually I'm left with just getting her out into a fairway with enough maneuvering space to spin her until I get pointed in the right direction.
If there's a considerable cross wind or current, I've got to weigh the possibility of becoming a "bumper boat" against my desire/need to get underway.
You just described backing our Cape Dory 28 to a tee.