This is not an easy question but it comes down to two main factors stretch and weight. Even the best sail making materials stretch. When they do the sail cloth alters shape. The more wind the more stretch. Minimizing stretch means that there will be less of a sail shape changed between the gusts (where the sail gets fuller and more powerful which is the opposite of what is desireable) and the lulls when the sail becomes flatter (which is also the opposite of what is desireable). Both over powering and blading out a sail can have major speed robbing impact.
Modern sail cloths are made so that they are stronger and less stretchy in one direction than the other and modern sail construction orients the fabric so that the strong dirction is aimed in a specific direction in order to minimize stretch.
The other problem is weight. If you did not have to sail in light air, sails could be built so heavy that they had negligable stretch, but a sail needs enough wind to blow it into a proper flying shape. In a moderate breeze most sails will assume the intended flying shape but as the wind gets lighter, gravity causes the sail to hang in a flat, non aerodynamic shape. The heavier the sail, the higher the windspeed that it takes to blow the sail into a proper flying shape.
You could make sails very light if you are willing to do frequent sail changes with the changes in windspeed, but for most of us we want a sail that can function in a wide range of winds and so the sailmaker tries to select fabrics and orientations that provide a proper balance between minimizing stretch and weight.