I have a 1980 Yanmar 3gmd. I believe same as the OP's engine but 3 cylinder. I think my boat is also over proped as if I run at anything over 2500 rpm the engine just goes faster but boat is already at hull speed so goes no faster. I do not blow black smoke, by the exhaust does become more gray and the engine seems to start running hotter and seems stressed.
I have also heard the yanmars like to run at 80% max rpm. But to me, that is really straining the engine. I know what Mack Boring and even the Yanmar manual says, but it just seems to be straining the engine, and higher rpm mean more wear on moving parts. I do however run the engine in neutral for 30 seconds or so at full rpm prior to shutting down- to blow out the carbon as the Yanmar manual says you should do.
I believe my engine has been run this way for the past 34 years and does not burn oil or have low compression. Exhaust is clear.
I would just run your engine at the lower rpm if that is what your boat is happy with.
The I also agree with the OP's PO- it is difficult to get the perfect prop specs considering the effects of wind and waves will have on the load the engine will see.
As far as the engine oil dip stick, sounds to me the your range is too wide, I would go to a yanmar supply and compare what they would have for your engine
my 2gm loved to run at 2600-2800rpm
yeah a smaller boat but was higher than most guys like to cruise at
for what its worth I do beleive that yanmars like to be loaded up a bit more
its something I have heard over and over again many places around...
also propping correctly should be done on a new or at least rebuilt engine since with a very old tired engine or simply run long hours wise you might be diagnosing and fixing symptoms caused by other problems and issues.
for example in this case it seemed to be the prop all along, but a tired engine with badly sealing or worn rings, and sticky valves or bad seating valves will exhibit the same symptoms
not to mention the getting hotter issue
which can be caused by improperly seating valves and carbon buildup
pistons that have a bunch of crap on the domes will create a fake compression and require better cleaner fuel in general terms...so if you find that all of a sudden(it takes years but you dont notice since performance degradation is gradual) you start getting hotter and billowing smoke and soot out the exhaust
BUT you have confirmed your exhaust, elbows, water lifts, manifold and all are clean and clear and not clogged up
that means you have to clean up carbon buildup and or start dooing at least a partial rebuild
anywhere from the fuel system to injectors to pumps
or valves, combustion chamber, pistons etc...
tolerances will get bigger as the engine wears
so a QUICK test is to add something like zddp additive by stp...and use a nice thicker oil than recomended
like someone else mentioned if this improves and decreases soot and exhaust smoke and smells, it will also quiet down the engine, it simply means your engine is a but tired in general terms and simply needs a rebuild
if you dont feel like doing this there are ways to clean up a bit without tearing apart but they arent always succesfull..
for example a high doseage in a full clean tank of fuel of SEAFOAM, under load at the dock or towing something for example, many times over can help to loosen deposits and stuff inside the engine
however the only real way of getting an engine back to specs is to get tolerances and parts back in new or original shape
bearings, pistons, rings, shims, valves, seats, flat surfaces, gaskets, o rings....etc etc etc...