We sailed all over the Med with an inflatable Avon ("Redcrest" model, I think but won't swear) that we pumped/deflated at each port of call. Almost never towed it, as it had a tendency to go airborne. It rowed pretty poorly, but we also had a removable transom and a British Seagull outboard for longer distances.
That was on a 38 footer with usually 2-4 able bodied guys on board to share the work, and no kids.
When my wife and I started cruising with our 3 kids, I wanted nothing to do with that arrangement. Too much hassle. With young kids aboard, our goal was to simplify/minimize as many tasks as possible. We ended up with a hard Dyer dinghy that we tow behind. No outboard, just oars.
We are on the Chesapeake, and towing has never been an issue. But I can see why you might want to avoid towing in areas with the potential for heavier seas. How has it worked with that Walker Bay? Presumably, you remove the outboard when towing? Is it the outboard that's the hassle? Not sure which WB you have, but 6 hp seems a bit on the large side. Could you downsize the engine and make it easier to heft it up?
An inflatable does offer the ability to shrink the package, with the associated work. Just brainstorming here, but I take it there's no place to store the WB on deck? How about a small, light weight pram, like the Eastport Pram
? Davits, maybe?