I'm toying with that same question myself. We have a 10' Avon with the inflatable V-bottom, but we perfer a lightweight motor too. Not sure if your dinghy is inflatable or what it's rated for--ours is 10HP max.
All that said, we're thinking about the 6HP Tohatsu (or similar). The reason is that it's the same weight as the 4HP and 5HP models, and only 14lbs heavier than the 3.5HP, but with the extended range of external gas tank and more power to push through a chop.
I don't know if we'll actually GET a 6HP--we might end up with something used or new on sale. However, our thinking right now is that it might be worth going from 41 lbs to 55 lbs but almost double the power. Just my two cents for now, and of course I might change my mind!
I have the 6HP Tohatsu for a 10'4" Achilles air floor. With one adult and a pair of kids, the 6HP will get the Achilles up on plane. With two adults, no planing. Getting the OB on and off the mother ship is interesting but do-able for one person. I always end up with a little bit of grease on my shoulder after doing it. The inflatable will take up to a 10HP, but there's no way I'd lug that sucker around except by car. If you don't have any need to entertain the kids by zipping around the harbor, then smaller might be the way to go.