Your concerns about roller furling are legitimate and it is something to consider. Generally speaking, a roller furling sail will be more effective (in terms of power) than a hanked on sail assuming it is completely unrolled. This is because the foil gives better shape to the leading edge than a hanked on sail would get. However, the minute you start rolling it up at all even good roller furling sails get terrible shape.
One of the questions in this thread regarded what rig is appropriate and I feel that one of the best rigs discussed is the cutter. The beauty of the cutter is that you can have a roller furling yankee and the minute it gets rough, you can roll it up and just use the staysail(you will have to reef the main as well). If it gets really bad, you can take down your staysail and hank on a storm sail.
It is interesting to look at ocean racing boats and how they are set up differently for single handed racing and crewed boats. The single handers do use roller furling headsails but they have a couple always there.
Regarding roller furling on the main, I am not a fan. If you have in mast furling, you add a lot of windage and weight aloft and can't have proper battens. Slab reefing gives better sail shape and is more reliable so that is what I choose.