DISCLAIMER: I do not own a Yanmar, not have I ever done anything other than bleed one.
Looking at your photos, and description, I suspect that the "machining" of the inside of the coupling was done on the boat. All it would take is very little play, and the shifting in and out of gear would start to hammer the inner spilnes of that coupling. Eventually, the hammering would shear off the splines, and with a little torque in the right direction, the nut would be spun off too.
I suggest that there may be little pieces of splines somewhere in your bilge.
To repair: get a new coupling - yours is clearly toast -, a new lock nut, AND a new washer (part #2 in the above illustration). I would go so far as to use lapping compound and Prussian Blue to mate the coupling and the output shaft. To get an idea of what I am talking about, look at this article by Maine Sail
pertaining to lap fitting a prop. Once the coupling and the output shaft have been prepared, use a torque wrench to apply the right amount of torque to the lock nut. Once torqued to spec, stake the nut.