Goof Off is not just xylene. Take a look at the MSDS and you'll find it is:
30-35% Diethylene glycol monomethyl ether
So yes, there is xylene in it, but that's like saying a human being is "just" water, because we have so much water content. In this case the other chemicals outweigh the xylene as well.
IIRC pure xylene kicks off and evaporates way faster, too.