I don;t think we have enough information to give this question a good answer.
A lot depends upon the type of fishing this boat was doing, were you passing in front of or behind, what kind of channel, etc.
Where is the location?
If he was lobstering, he probably over-reacted.
If he was bottom dragging, maybe not, depending on the channel, etc.
My guess was he might have been gillnetting or purse seining.
How big was the boat, what did it look like, was it alone, or was there a small working skiff near it?
Lots of variables, and I'm not talking about COLREG here, but about getting along and sharing the resource.
Certainly some comm. fishermen are jerks when it comes to rec boaters, esp, sailors; but not all, or even most.
This might be the start of a lively discussion.