Re: who had right away?
Personally,I think you we're in the right, to avoid a collision is the underlying truth no matter what the other guy is doing or not doing no matter what the Coregs say.
Commercial fishing vessels do have the right of way, but only while actively engaged in fishing, to me it sounds as though this situation was different.
Next time, avoid the situation, try to hail them on the radio, and if no response, video them or get their description, type of vessel, location and any numbers you may see, and report them as making an unsafe passage to the CG. They may be able to help.