John, you need to work down those lists of possible causes, one by one, and make sure everything is optimized to make the boat fast.
Is your hull or keel dirty? Are your sails blown out? If there's no plastic coating left on them and they don't feel scratchy, odds are they should be used as bedsheets, not sails. Then there's your rigging. Mast, shrouds, stays, all have to be tuned. If your spreaders aren't original (or even if they are) they may be too long. Stock J/24s used to come with spreaders that could be cut down several inches--and the guys who pointed higher always cut them down so the genoa could be hauled in.
Sail trim, weight distribution, crew weight, all can only help so much, but first you need to check out the rest. That can mean someone from a local loft, or someone with more experience, you do what you can. And if you bought the boat used with anonymous sails on it--sometimes they're simply the wrong sails for the boat. You may need a tape measure to check.
It is like playing "where's waldo?". Look carefully, you'll find him.