Funny, I always took that as the row boat/kayak etc was a boat maneuvering under power - albeit not a power driven vessel which specifically is spelled as a driven by machinery - and as far more maneuverable than a sailboat as it can steer in the full 360 degrees.
I saw nothing in rules 1 through 18 to disavow that opinion.
Not to mention, much like a freighter, if I hit one I won't be the one bleeding afterwards (i.e., right of way = might of way)
I think what it comes down to is if they don't move, you must take all possible and reasonable actions to avoid a collision. Then drop sails and yell at the idiot on the loud hailer
Common courtesy says the rowing teams etc.. including races should be given right of way much like we'd want on the race course. I really hate it when a non-race participant come sailing through a race course on starboard and makes even one turn off course.