Nope. Rudder goes to leeward (away from the new windward side). It's the tiller which goes to windward. The rudder is turned this way to avoid any possibility of tacking.
It's the mainsail which keeps the bow up, not the rudder.
Because you're moving only a bit, the rudder has much less effect than when sailing. You have to experiment with your boat to find the right setting of the rudder, but on many boats it's full to leeward.
Mark, if you're "spinning" it sounds like you might not have the rudder positioned correctly - e.g., your rudder isn't counteracting the tack.