I feel your pain!
I too have great difficulty in backing the boat up (same boat/prop set up).
I would agree with the idea of starting with the rudder centered, the boat aligned off center before starting (to allow for prop walk), and using neutral as much as possible. From your skewed starting position, turn the prop only long enough to be making way. Go to neutral and only use small rudder movements. I agree with previous statements that any more than half over on the rudder and it acts as a brake, not a turning device. To help with this, I marked my wheel at the center position, and don't turn it more than 3/4 turn either way. In theory you can use a quick burst of high throttle in forward (turn to port) or reverse (turn to starboard) to "spin" the boat with the prop walk without changing speed too much. I have not mastered this at all!!
Again, as others have said, practice, practice, practice.
Good luck, and if you find something that works, please share it with us.