Like many assumptions, it is a little off the mark. It wasn't actually a purse (I was trying to keep my wording simple) - it was her canvas beach bag that held hats, sunscreen, snorkels, etc., and was sitting in the cockpit the whole time. The GPS signal was perfectly fine - 8 or more satellites at all times. This GPS is extremely sensitive, and even gets a reliable signal in the middle of my house. I have no reason to doubt the reliability of the tracks.
Even if the track was a little off, it was exactly the same track on all three examples that I showed above, which means that the disagreement between the three charts can only be attributed to the charts. If you look at the track from the whole day trip, where we passed close by Cooper, Norman, Peter Islands, etc., it is clear that the NGA chart was significantly more accurate. The Garmin and NOAA charts showed us going over ground on numerous occasions, and the NGA chart agreed completely with our DR positions.
I believe that NGA makes the intelligence maps that guide our Navy vessels and cruise missiles, so I'm betting on them to have the best chart.