The Mercury engines under 30 hp are made by Tohatsu in Japan and I compared my 5hp Merc 4-stroke to a 5hp Tohatsu 4-stroke last summer and they are indeed identical. I believe the Nissan is also the same. My hull weighs approx 2400 lbs loaded and the 5hp is very adequate and hits hull speed. My son borrowed the motor while his Honda 8hp was being rebuilt on his 30' Catalina and it did not reach full rpms with the 5hp Tohatsu. It clearly seemed underpowered on that hull.
According to Mercury the difference in the 4/5/6hp engines is not just jetting, the carb itself is constructed differently on the inside. The props can also be slightly different. I have been toying with the idea of buying a 6hp carb just to try it out.
I think you have a case of too much boat for a 1-cylinder engine. When you're talking displacement hull such as a heavy sailboat the number of cylinders makes more of a difference than hp which also helps to explain why a 4hp 2-cylinder performs better than a higher hp 1-cylinder. My Mercury 5hp vibrates considerably more than the old 8hp 2-stroke Tohatsu I had. Mercury and the dealer both say it is operating normally. I've gotten used to it. I would say it is quieter than a 2-stroke though.
You might check the valve lash as the poster suggested above. This could definitely affect performance. If your boat is 3000+ lbs plus gear I would say that that is more likely your problem and you need to upgrade to a 2-cylinder engine either 2-stroke or 4-stroke. From your original post I thought your speed over ground difference would have been greater ... I don't think you're that far off and I'd bet that a 2-cylinder engine will be night and day difference.
I too think I have to much boat for the engine. It is in my opinion a very weak 6hp.. My boat is only 23.5 feet long. a very very substantial difference from a 30 ft boat. As I have said before I would not think much of it except that I have run a 4hp engine given its a 2 stroke and it ran stronger than the new engine. regardless of 2 stroke or 4 stroke I purchased the engine to move up from my current engine in power. The manufactures rate these things in hp. It is rated at 2 hp 50% more than my old engine and is weaker in pushing the boat than the 4hp engine. If nothing is wrong with the engine then it leaves me feeling as if the manufacture is playing some kind of numbers game with me. Misrepresenting their product. The biggest thing to me is that the engine on the water at full throttle misses max rpm by around 1400 rpm and nominal by around 900 rpm.. on a engine that is supposed to do 5500 rpm to put out 6 hp missing 900 rpm is about 20% of its rpm range. I would guess it is more than 20% of its power missing.
When some poor slob like me goes down to purchase a new engine for our boat we look at the one on it now and go.. hmm I want more.. If I have 4hp now I want more power.. 6 hp should be 50% more. Ok I will get that 6hp 4stroke.. they even have it specifically made for sailboats with 20 and 25 inch long shaft. cool.!!! so poor slob pays his money and orders it, gets it in.. diligently breaks it in and then takes it out and opens it up and .... ummm .ummm well its.. hmmm.. slower than the old engine?
Spend 2 months being polite and working your butt off for the manufacture to figure out why... to be told.. well its just not enough engine for your boat.. you obviously should have purchased a bigger engine.
hmmm... i thought I had? 50% bigger as a matter of fact.
I was semi clueless prior to this episode. I have educated myself a lot during it. It is a complex subject with what prop you have on it making a lot of difference, also the torque differences of 2 stroke vs 4 stroke making a difference. I think now that the manufactures or at least tohatsu in this case goes out of their way to play down the actual comparative performance of the engines.
When someone really cranks down and asks why... well its your fault.. your boat is to heavy, you must have stuff growing on the bottom (not), your engine mount needs to be moved up. They will not address comparative performance at all. And when asked why the engine will not produce nominal rpm they just keep saying the boat is to heavy. If the engine was producing 5500 rpm while pushing my boat with these props and it was going slow then the first thing I would do is jump in the water and look for the anchor I'm dragging.
I'm sure it is obvious i am unhappy with tohatsu. I believe that these engines are weak, not that they don't run or aren't reliable, just that they are over reported for what power they produce. I believe that tohatsu knows this and seriously downplays it.
I like the way the trolling motor company's rate their products in lbs thrust. I would love to see outboard manufactures do the same. They could rate the thrust for each engine with each of the prop choices available. I think this would introduce some meaningful numbers for end users. I don't think that the hp ratings now on the engines are meaningful numbers giving my experience with this engine.