Am I wrong when I think that the further back the pole gets, the further off the wind/more downwind/slower you are getting? How does this help VMG?
That's right: the deeper downwind you sail, the slower you go (as lift decreases), and the further back you bring the pole, to a maximum of about 45°. The slower speed is offset, however, by a more direct course sailed to the mark. The J/29 I race in BC is a good example: on a light wind day, say under 12kn (90% of our sailing weather in BC!), we head quite steeply downhill, and slowly but surely win the downwind leg, as the asym boats criss and cross: without enough wind to plane well and go really fast, they can't bring the apparent wind in close enough to head down deep, and have to reach back and forth to the leeward mark. We'll be doing 7 knots, but the boats flying asymmetrical spinnakers sail twice the distance and only do two or three knots more of speed on average. In higher winds, then everything flips: we'll top out downwind at 9 knots (maybe 10-11 on a surf), and the asym racers are all planing at 12+, and can sail deeper as they get their apparent wind forward. The fastest boats that sail in this manner (some cats and tris come to mind) effectively sail upwind on BOTH windward and leeward legs! This is why they forgo spinnakers altogether and use Code 0s.
As a general rule, for windward/leeward racing, displacement hulls are designed to use symmetrical kites, and planing or semi-planing hulls are more successful with asymmetrical kites. Some, but not many, use both types, depending on conditions: the PHRF penalty is significant, however. Needless to say, triangular courses favour planing boats with asymmetrical kites, and they'll handily pass the older boats on the reaching leg. Offshore, asym boats also typically do well, if you find yourself reaching for any significant part of the race.