dont know the area but sounds as if there may have been an element of wind against tide causing steeper seas with a tendancy to break. definetly better to sail than motor far more comfortable in rough water.K
The far side of Catalina is pretty much open ocean, there really aren't any tidal currents to speak of (unless one is REALLY close to shore). However, the fact that the wind increased in intensity rather quickly probably accounts for the steepness of the wavefronts. Without much fetch (distance AND time), the waves just hadn't had enough time to sort themselves out.
First of all, you made the right decision, for you and your crew. Had you kept going it would have taken you several uncomfortable (and anxious) hours to get to Avalon; turning back to Cat Harbor may have made for an even less comfortable ride, but undoubtedly lowered your anxiety level considerably (since you knew that a safe harbor was just a little ways away).
Given your description, it sounds like the wind was Force 5 - gusting to Force 6 (I never have had a boat with a dependable anemometer, so I just use the good ol' Beaufort scale). At that wind speed you can get fairly large waves IF the wind has been blowing for several hours (maybe 8 - 12) over open water. At the start of a blow the waves will be considerably smaller, BUT much steeper, often making for a very uncomfortable motion. Once you get some more experience you'll probably consider conditions like this "spirited sailing". However, Force 6 is about as strong a wind with which most recreational sailors feel really comfortable.
The point has been made that motoring into the wind and waves probably made for a less comfortable ride than running with them. That's true for three reasons: first, your apparent wind probably increased by 12 to 15 kts (your boat's speed x2) when you turned around; second, you were also plowing into the waves at a much higher speed (wave speed plus boat speed, rather than wave speed minus boat speed), making the impact of the bow against the waves much more forceful; and lastly, the face of a wind-driven wave is much steeper than the back of the wave (unlike the situation for ocean swell, where the two sides are more or less symmetric). Together, all this probably made for a pretty wet ride back to Cat Harbor.
But, at the time, I'll bet you felt better turning around. Why? Probably because you had a near-by harbor to aim for (always a comforting thought); and you weren't used to the yawing that a following sea will induce. With more experience you'll learn how to limit such yawing (by a bit of judicious use of your rudder, and by sailing with the wave off you stern quarter), and you'll get used to the feeling of a small amount of yaw.
But, don't sweat it too much. NO ONE sails for any length of time without getting a bit scared now and again. Remember, that which does not kill us, makes for a great story to tell (and, the story will only improve with age, since each time you tell it the wind will be a bit stronger and the waves a bit bigger