The description of the situation may bear clarification, but it sounds like boat A overstood the mark and is footing off towards it, and that boat B tacked on the layline to leeward of boat A. Boat A, moving faster, is passing boat B when boat B luffs up. You appear to say that boat A responded to B's luff by turning away (heading up, as would be expected) which brought her stern within 5 feet of boat B.
Boat B should have expected boat A to head up as a result of its action, and therefore to have boat A's stern swing closer. Though it depends on a number of variables, like wind and sea conditons, if boat A's stern was 5' awar from boat B at the closest point, it sounds like boat B overreacted in calling it a foul. If boat A moved clear ahead of boat B within a few seconds of boat B's luff, and the closest the two boats got was 5', then it would seem that appear that there was no foul. Boat B simply got passed, and didn't like it.
The distance at which one must begin altering course to avoid contact depends on many things. The speed of the boats. The wind direction and speed. The point of sail of the boats. The sea state. The caliber of the skippers and crews... and probably twenty other things you could think of. Catamarans reaching at 25 knots might have to begin reacting to a competitor's moves a quarter mile away in order to avoid a foul. J/24's in light air might get down to a couple of feet. Dinghies at mark roundings work in inches. The rules also mention manoeuvering in a "seamanlike" manner in several places, and this may be one of them. Check the Racing Rules of Sailing (RRS) on the ISAF website: ISAF : Home page
According to your description, boat B tacked below boat A. If boat B had tacked on top of boat A, boat A would be to leeward, and boat A would have had ROW. There are rules about how this applies wtih boats on the same tack passing one another within 3 boatlengths of each other, and how quickly the leeward boat can luff up. Check the RRS on this too.
A tack is considered completed, and the vessel gains ROW once the bow passes head-to-wind, IIRC. (Look in the RRS again.) There are limitations on the ROW vessel's actions which require them to provide time and sometimes room for the burdened vessel to respond. You can't slide up to leeward of someone, throw your tiller over to leeward, and then claim they didn't respond: you have to give them time to respond in a seamanlike manner. If they don't, THEN you have grounds for a protest.
It looks like no foul to me, but then, I wasn't there, and the description of the events seems a bit fuzzy in a couple of places.