The more I think about the intermittent nature of the problem the more I think it may be an injector problem. When knocking and putting out excess smoke one of the injectors may not be producing a proper spray pattern, due to debris or water?
If it was fuel starvation I think you would just lose power. Kind of sounds like fuel may be puddling up? Are there any indications that excessive blow by vapors can be collecting in or near the intake tract?
Having the injectors checked by a shop would be easier than pulling the head?
James' comment about cetane additives would be an easy try. Or maybe a heavy duty "cleaner" additive? I used to use this in my VW diesels & never had any fuel related problems:
DIESEL KLEEN® +Cetane Boost®: Max HP Formula ? use during non-winter months for the ultimate in performance.
I am sure there are lots of different additives that may be as good or better?
Paul T.
Edit: Just saw your picture, the tanlk has been around for a while.
I did some tests with the Power Service additive a long time ago. Put some diesel in a coffee can, added a little water to it, then added the additive & the water was no longer visible. A tank that old may very well have some water in it. However, a good filter/separator should catch it. Some units have a clear bowl on the unit allowing you to see water or debris.
Paul T