Great trip. The first time I did that (7 times, now) it was just me and my 8 year old daughter. Fantastic.
Yes, it is a really bad idea to begin any coastal passage in weather sufficient that you might have to bail out. However, there is at least one nearly all weather entrance for a boat you size; Chincoteague. I've entered when there was a 12-foot on-shore swell running. It does not break, like Ocean City can. Do NOT be put off by the fact that the charts show the approach as variable; there is a large CG station in Chincoteague and a commercial fishing fleet (big draggers); they keep it well marked. That said, Ocean City and Wachapreague can be casual if the weather is not on-shore and your draft not too great.
Read my blog, searching Delmarva and Trip Reports. I think you will see that we've had many trips just as you describe. Never an over night, breaks a the ocean here and there (Chincoteague is great for that).
Sail Delmarva: Trip Report - 2010 Delmarva Circumnavigation