Ask the "fisherman" to show you his fishing lights (e.g. green over white...etc). If he's not rigged to show fishing lights / day shapes, then he's not a fishing boat.
Well, this has all been interesting reading. With the exception of small sports fishing boats, any boat that looks like a commercial fishing vessel with things like pointy outrigger poles, big net reels,all kinds of lines running all over the place, and possibly needing a paint job, and going slowly in a straight line, is probably "fishing", regardless of what may, or not, be hanging in the rigging.
Back to the original post, the skipper of a commercial charter boat trolling with side planers? out probably thinks everybody should get out of his way because he is more important than just about everybody else, except maybe somebody bigger than him?
I guess one should do the best one can to avoid a collision or putting another vessel up on the rocks.