Thanks for the ideas. Here's some answers to the questions:
I changed the oil myself and used Delo 400 LE 15W40 -- the same grade as listed on my last oil analysis in December (my user manual calls for SAE30HD/10W40). I checked with the P/O about overfilling. He used to keep the level at the dipstick's high-mark (I refilled part-way between low and high). I think this pretty much rules out overfilling as a possibility. (The reason I thought is might be relevant is because my mechanic flipped out when I told him that I though my engine takes just shy of 3 gallons of oil -- which it does.)
There's nothing loose on the exterior of the engine as far as I can see (I have good access all around). I get the slapping noise whether in gear or not. RPM seems not to be a factor (almost no sound while idling, noticeable at >1000rpm and louder as the rpms increase). I don't remember load being a factor (noise is present whether in neutral, forward or reverse.) As far as I can tell the noise seems to come from the top of the engine. I'll try the screwdriver in the ear trick and see how I get on.
I can't say for sure when the problems started -- around the time of the last oil change (I have 8 hours since the last change). I also refueled around that time too. It definitely seemed to be getting louder over time -- its what prompted me to get a mechanic. Another possibility: when I replaced by Racor filter (quite a long while ago), I topped off with diesel from a gas station (bad diesel? should have strained the fuel I used?).
My mechanic found that one of the injectors had failed and the others were in bad shape. He replaced all of the nozzles and one of the fuel lines, but the slapping remains. I wish he'd done a compression test. Unfortunately he didn't think it was important. I have a friend who has a kit. We may be able to do a test if we can get the correct adapter.
Currently the exhaust is mostly white. Looks like there may be some oil in it. It thins out quite a lot when the engine warms up. Mostly it smells quite heavy -- like there a lot of fuel in it.
Last weekend we took the rocker cover off and measured the valve clearances. We found that some of them were 3x the clearances specified by the manual. Reducing the clearances didn't seem to do much to reduce the noise. Rapping on the top of the valves seemed to reduce the clearances too (suggests knocking off carbon buildup?). We're a little nervous about reducing the clearances much (if carbon buildup is the problem, and more carbon falls off, there may be no clearance left - could cause valves to stay open. Could this cause permanent damage?).
We also found there was a little blow-by (but no oil in the re-circulated air).
Next weekend, I'm planning on changing the secondary fuel filter and making the valve clearance to spec to see if that help.
I've also found a fuel additive at Kragen that claims to help remove carbon buildup in diesel engines. Anyone ever tried this?
Cheers,
- Stu