There's a tad of ambiguity in your (original) post so pardon me for wandering.
3-4 = knots or Beaufort Scale?
Assuming 3-4 knts of wind speed upwind you can get more speed (in a typical 33 ft cruiser, maybe 2.5 - 3 knts VOG) in such light air because the apparent wind will be higher and that will help, you say you have owned several boats so I'll assume you know that.
Wind off the beam, and especially behind the beam with no apparent wind to lift you, of course you will 'slow right down' - the best you make is wind speed minus drag - probably all of 2 knots tops in a typical production boat.
Then again, with wind that low and (for us readers) unknown current - the act of changing from upwind to offwind could mean the current/tide caused the boat speed change.
It's really hard to guess all the factors - so I mention them in passing, no insults intended if it seems a simplistic response.
You don't mention your jib or what it is doing so I'm assuming it's set correctly and wind ahead of beam telltales are flowing aft, wind behind beam the forward telltales are limp, inside are dancing independentally.
Luffing the back of main with the wind on the beam implies the front of the main is full - which means the draft of the sail is moved forward by releasing the sheet more than you feel is appropriate, yet not enough to luff the front of the main, admittedly strange, but why your boat speed improved - your forward driving force improved. You can get the same effect (draft shifted forward) by leaving be on the sheet and easing your outhaul and creating a fuller mainsail (which is the correct thing to do in light air). Leech line tension should also be checked as others have indicated, it should not cup and trap air, it should let air flow smoothly off the back of the sail while still remaining 'attached' to both sides. Incidently, in air that light it also helps to ease the main's halyard just a bit and fall off a tad more than normal when going to wind.
If I got anything screwy I assure it because of multiple edits