It's all on you and your comfort level and or experience.
In my opinion, nobody can answer this question other than you.
Some say it is never too rough; others with little experince don't like 20knot winds or 4 foot seas. Don't let anybody talk you into doing something you are not comfortable with. Over time and with experience you will get comfortable in different conditions.
And when you do find yourself on the edge, always remember, "The boat can take more than the crew."
Courtney is My Hero
If a man is to be obsessed by something, I suppose a boat is as good as anything, perhaps a bit better than most - E.B. White