"Help" can mean a lot of things. Sometimes it can mean simply placing a phone call for one of these grounded people. I'll bet that plenty of them set out without a working VHF and a cell phone with only half a charge.
If someone is in genuine distress and signalling appropriately and you witness it, you are obligated to help, but not necessarily to place yourself at risk. Make a phone or radio call on their behalf, and let them know that you did so. Ask if they have any medical issues. Pass them a first aid kit, if capable of doing so, or inform the CG or local marine patrol if you are not.
For the most part though, it sounds more like you're just not interested in helping.
On one hand, I sympathize with your motivations. There are a lot of idiots on the water, and you're not SeaTow. You're not interested in burning out your little outboard engine, yanking someone out of the mud, or risking damage to your boat, and spending all saving people from themselves.
On the other hand, you yourself have accepted a tow from a skiff that may have had the same concerns. You might consider helping out occasionally, so that you can put a few points in "Vigor's Black Box". It may help you in the future.
John Vigor's Black Box Theory