Cupped or Fluttering leech?
the leech of all sails will, at some time, flutter, and the only way to do away with the fluttering is to pull on the leech line which makes that edge cup. Even mainsails will have this problem except that the battens prevent to much cupping or fluttering.
Heres the reason...Stretch. The sail cloth at the leech takes more load and abuse (tacking agaist the rig) than anywhere else on the sail. The stretching of the sails makes the local leech longer than what was designed. This usually happens after some time but if it happens right away with a new sail then that sail is under built for the conditions it is being used. The more stretchy the material the sooner it will happen and larger the affected area will be.
The only remedy is to either live with it or have your sailmaker re-cut the leech and ask if they can add any strength to the leech by adding a leech ply or some other second layer.
With new sails be sure that the salesman or sailmaker does not under build your sail, AND, be sure that YOU do not use the sail outside of it's intended range.