Laura, you have no idea how good Dave's advice is. Maybe either you or your husband could stay on land with the kids while the other sails the boat with an experienced delivery captain down to Baltimore, or better yet, Norfolk.
The boat would be given a good shakedown on the trip, and at least one of you would gain a lot of sailing experience. Once the boat is in Norfolk, you would be in the company of a lot of other boats doing the icw trip. I bet you could even find a some folks to buddy boat with down the icw.
I won't say that the trip is cake once you are in the icw itself, but I know I would feel a lot safer if I were in your shoes starting out in Norfolk with all the family aboard.
I would question the sanity of a delivery captain who would do the trip from Maine to Norfolk with all of you, including the newborn aboard.