I like sailing downwind with just the genoa. It pulls the boat along and gives me a more stable feel. My main also blankets the genoa, and reduces its efficiency (unless going wing and wing). It's easy and relaxing to maintain just one sail out.
Going upwind with just the genoa or jib is a bad idea because it will "push" the bow away from the wind. That is the reason you will have the helm pressure that Zanshin is talking about. You will have to work harder to stay on course. The main will balance that "pushing the bow off the wind" effect.
Imagine a picture of your boat looking down from directly above. Find the central point of the boat (relatively). This is like the pivot point when your sails are up. Imagine just the headsail up. The wind will fill that sail and force the boat to turn on that central pivot point. You have to adjust your steering to compensate for that force on the headsail.
Try going upwind with just the jib some day and you'll feel the unbalanced effect. I wouldn't say it would damage anything, just hard to sail that way.
Dictated, but not read.