Really depends on a number of factors.. the area, depth, holding, individuals' comfort levels..
Faster missed something: Insurance paid up?
But seriously, in the situation you described I would have gone below, mixed a drink, sipped it slowly in the cockpit while I assessed the situation and how the boats were tending. In the situation you described, nothing much would have changed and so I'd finish the drink, go below for dinner and then have good night's sleep.
In the situation you described it seems to me the guy was a jerk or inexperienced, or both. Anchoring rights are important, and I defer to those anchored before me when it's obvious that there is a reason to be concerned. What you're doing when you ask the boat anchored CLOSE to you is: 1. being polite; and, 2. seeking the judgement of another skipper. If you're close enough to speak to each other in a normal tone of voice, you should pay attention to his views, as he was there first. If you have to shout to hear each other.....well, go below and mix a drink.
As always when at sea, what you do is ultimately up to you and, of course, you accept responsibility for your decisions.