If boat B was on her layline and not close-hauled (correct me if I misinterpret), then by luffing up to "create a penalty situation", she is sailing above her proper course. A does the right thing by altering course, but has grounds to protest B's luff. At that point B should sail clear and do her penalty turns, so the fact that she had to alter course is irrelevant.
That's how I see it.
Btw, more experienced racers tell me that skippers rarely win when they protest sailing above the proper course, because it's so hard to prove. Still, between that and Paul's well-made points, sounds like A did everything right. There's no rule I know of that says you can't get close to somebody (and if there is, it's the bit about permitting the other boat to avoid you in a seamanlike way, which B does not do).
s/v Laelia - 1978 Pearson 365 ketch
s/v Essorant - 1972 Catalina 27