The COLREGS, as found on the CG auxiliary site (http://www.auxetrain.org/colregs.html) define as follows:
[The term "vessel engaged in fishing" means any vessel fishing with nets, lines, trawls, or other fishing apparatus which restrict maneuverability, but does not include a vessel fishing with trolling lines or other fishing apparatus which do not restrict manageability.]
The list of "pecking order" is correct with exceptions (such as Jeff H pointed out) of: narrow channels, traffic separation schemes and overtaking.
To answer your question, I don''t assert my "rights" while under sail to interfere with any other boat that is fishing or crabbing, even when the COLREGS define them as having no special rights. I haven''t encountered a situation (in my admittedly little time on the water) where my detour around a drifting sport fisherman has been a big burden. I consider it a "courtesy thing."