Your fishing friends sound like the truckers on the interstate. They are working so the cars had better get out of their way!!!!
With this logic a man driving home from a hard day at work has special rights to run cars filled with people going to dinner off of the road!!! Heaven help someone walking or on a bike!!!!
Actually I have no experience with commercial fishermen. I have met a few powerboaters with an attitude. I do what it takes to avoid a collison even when I''m clearly the stand on vessel. But as I understand the rules you are also liable if you cause an accident by not standing on when you are the burdened vessel.