What were the currents doing at the time? What you describe sounds (in my limited experience) like a strong current-against-wind situation, pretty common around here. My feeling about closely spaced, steep waves is that you've got to keep some drive for control and balance on a beat, and that heeling is better than being tossed about. I would say you did the right thing keeping up a scrap of main. Can you reef your staysail?
s/v Laelia - 1978 Pearson 365 ketch
s/v Essorant - 1972 Catalina 27