I apply one fresh top coat every season. It keeps growth down and I can easily get away with a hull cleaning every 2 to 3 months, sometimes 4. Granted, we are in a relatively lower growth region than southern climates. I went one full season from April 1 to Oct 31 with zero cleaning on a fresh paint job. It was slimy, but zero hard growth (other than the prop and shaft and it did kill one knot of boat speed by the end of the season). Long story why this happened, it wasn't planned.
If you're not racing, I don't think a little slime makes much practical difference. A full season's worth does. Otherwise, poor sail trim and blown sails are much more detrimental to speed than a couple of months worth of growth. FastBottoms, who posts here, has made a good argument that infrequent cleanings require more abrasive tools, which can reduce the lifespan of your paint. I'm sure that is true, unless you top coat every year anyway.
My biggest gripe with slime? It looks terrible when you're heeled over and blowing past another boat.