Not sure why you woldn't just motor the 4 miles. Thats not very far. Nice sail over fill up and sail back, no big deal as far as I'm concerned. Second of all, How often are you really going to need to fill up? Sounds like your more into day sailing than cruising, if thats the case, a full tank of desiel should last a long time. Depending on your tankage and if you are doing mostly day sails, I would think that a full tank would last several weeks, maybe more. But you'll have to figure that out yourself.
Also, your marina may frown apon you filling your boat with jerry cans at the dock. Most marinas are very enviornmently friendly these days and they don't want you spilling diesel into the water. And remember your responsible for any large spills.