Won't you be heading upwind to hoist your main anyway? You can hoist the
jib immediately afterwards.
If you're thinking of using only the
jib, well, it might be possible to unfurl the
jib downwind in 5 knots but at 10 knots there might be too much pressure in the sail. Having the main up would negate this problem since you could unfurl the
jib in the lee of the main.
In the end, in such conditions you should be able to head upwind and douse the jib in under a minute.
I have done what you're suggesting (except with a hank-on drifter, very light cloth, 155% LP) in very light conditions (just a couple of knots pretty far aft) for a short trip. I knew the wind was so light that the main wouldn't even fill. All the main would have done would be to shadow the jib and get damp. The trip was slow, but so was the wind, and we were in no hurry.
Jeff's implicit suggestion is that you sail two reaches, one on either tack. This will increase your apparent wind, and even though the trip will be about 40% longer, you will probably go more than 40% faster.