If the wind is up, which it sounds like it was, I would drop the jib before I reefed the main. Going with main and staysail, you can still point nearly as well as you will with full sail, if you have a decent breeze. And if the jib is old and fat, you will point better without it and it will be windage forward if it's flogging, pushing the bow downwind, toward the reef.
It's hard to resist the urge, but you can't chase wind angle with a full keel boat. That's fools gold, if you are going slow, you're not really pointing as high as you think. There is too much leeway if you don't have any speed through the water. I would maybe try to build up some speed off the wind a little, hopefully not giving up too much ground, then pinch up and let the large mass of the boat power through a wave or two, then fall off again and build up speed and repeat. That would be my best guess, boat speed off the wind and momentum, but you have to keep boat speed up or you are going sideways.
Not trying to second guess, just going thinking about what I would do in that situation, since I have the same boat and have a pretty good feel for how it sails. What you describe is a tough situation in any boat. Love the solution to getting the engine started, very clever. Years of problem solving in front of a screen may have saved your skin that day.
Damn, and just when I thought there was no greater piece of crap sailing to weather, than an Island Trader 38... (grin, bigtime) Just kidding, of course - I was favorably impressed by the sailing ability of the 45 I ran, considering the type of boat it is...
One of the things the Downeaster 38 is known for, is being underpowered... The standard factory install of a Faryman 24 Hp is woefully inadequate for that boat, IMHO, and it's still a bit surprising to hear of a situation where such a small engine made such a big difference... (assuming SERENITY had that original powerplant, of course) Presumably, Glenn had been making progress to weather to put himself in that position to begin with, and I understand there can be some current in the vicinity of Cedros... More than anything, this should be a cautionary lesson in the avoidance of a lee shore situation in an un-weatherly boat to begin with - but not having been there, and not having read his book, that's all conjecture on my part, of course...
Your suggestion to go with the staysail and main alone in such a situation, however, with a boat with a bowsprit and so much freeboard forward, is right on the money... That likely would have offered his best way out of that box, the much tighter sheeting angle and flatter cut of most staysails might have afforded much better progress to weather... Last year I ran a Valiant 42 back north from the Rio Dulce, and we were hard on a good NE breeze all the way up to Isla Mujeres, and then across the Yucutan... Not only did we make a roughly equivalent VMG under staysail and main alone, than when trying to carry the jib, it was a more comfortable ride as well, less punishing to the boat, and much less water coming over the deck...
Kudos to Glenn for his creative solution with the engine, but this is another example of the value of a cruising boat being "nimble", and weatherly, and being able to sail out of a tight spot... There have been a couple of threads here recently re windward performance, and I'm always a bit surprised to hear some dismiss the importance of such ability in a cruising boat... Engines so often fail at the absolute most inopportune moment, and one may not always have the time to futz around with McGyver-style fixes, you need to be able to
sail your way to safety, if anchoring is not an option... I think it's not uncommon, that people get into trouble by wasting precious time attempting to get an engine running again in a dire circumstance, when the better initial response would be simply to sail your way out of danger...
And to Glenn, nice to see you show up here... Please understand, despite my initial post to this thread, I do not consider your trip to be a "failure" because you abandoned your plans to sail across the Pacific, and beyond. Obviously, in your case, you made the correct decision, for you... Again, your story should simply serve as a cautionary, informative tale in regards to The Reality not always matching The Dream, and the advisability of starting out cruising by taking an approach more akin to baby steps, than by jumping in with both feet... For some people, like the late Mike Harker, the latter approach worked out great, but it's certainly not a guarantee for everyone...
Best of luck with the book, and whatever path you choose next...