In our 37 Beneteau, we make this trip about twice a month, during the sailing season. It takes us, directly, five hours; with some, or all sailing, depending on the wind, up to seven hours.
Don't let the Patapsco River tide discourage you -- it can be very strong. Look at the Tide Location Selection for Maryland
to get the tides on your side. Going back will be better, no matter which way the tide is going.
There is a small anchorage between Baltimore Marine Center and Anchorage Marina. Anchorage Marina is where we keep our boat, and, you may find a little more comfortable to transient.