I would hope that the "Port Port" was someone on the other boat's crew addressing the helmsman.
The only problem with non racing boats in a race course is that non racers may react differently than racers in the same situation. A Port tack boat can cross in front of a stbd tack boat and will often cross with less than three feet between his transom and the bow of the stbd boat. In a race typically the Stbd boat knows this is going to happen and is prepared for it and holds course. If someone not accustomed to this starts taking evasive action by steering low at last seconds they might in fact make an action that could cause a collision. That is probably the only difference between racing and non racing boats,
Another point. With 300 feet of warning it would have been very easy for you to steer a course that would have avoided this situation altogether. yes you did have the classic Stbd-Port rights over the other boats but why push it when it does nothing to help you? Also being on auto pilot in such a situation seems silly. 2 -3 seconds to disengage is 2 -3 seconds you may not have.
Sorry to be blunt. You had right of way but probably should have steered clear long before the incident occurred.