You asked that I comment in this thread, however, I don't know if my experience is universal to make a difference. I will add my comments from our recent trip... Just one man's comments about one specific trip.
My comments apply to ONLY the Canadian side as we did not visit the USA on this trip. There is a large difference in geography and availability of ports on either side.
We bought our boat in Bayfield; The river wasn't dredged until the end of May and we were one of the first boats to go in. Up-river, we were plopped into the water, backed out of the slip and promptly ran aground. With a 4' draft, there was only 3'11" depth. The marina operator said, "ohhhh, we didn't dredge there!" *sigh* I moved slips immediately and moved to Goderich the following weekend. The point being that rivers silt up quickly and you will be at the mercy of the dredge in any river port.
Moving north, I had no problems with a 4' draft. There were much larger vessels in every port and anchorage all the way up to Killarney and the North Channel portions we visited. You must be careful of your depth on the chart, as Lake Huron/Georgian Bay are much lower now than you'd expect - but that's just a matter of watching your charts and depth sounder, not of the draft of your boat. We saw plenty enough 50+' sailboats that quite easily had a draft over 6' so you should be fine with any of the 30'-40'-ish boats you were mentioning before.
Returning down the lake you begin to run into problems south of Bayfield or Grand Bend. There isn't much there for you to tuck into on any sort of sailboat... again, not a draft problem, but one of geography... there's no good spots for sailboats until you get to Sarnia. Again, no draft issues in Sarnia or rivers going south to Lake Erie due to the freighter channel.
*ahhhh* But Lake Erie. Lake Erie was above Chart Datum in a healthy way when we came through. However, this is where we began getting advice of "Don't go into Port _______ " (Fill in the blank). Many of the eastern ports are river based and have silted up with no dredging having been done recently so there are definite problems with depth. Even Port Stanley, the 2005 Richardson's charts show a straight channel exiting Kettle Creek, but in reality, they've had to make an S-chicane to mark the channel leaving the main harbour... fortunately, we were floating high enough over it, but you could see the distinct colour of the bottom and my wife and I knew we had better keep a lookout.
Further east, we were invited to a fellow's place in Port Burwell. He has a Tartan 34 which he keeps there quite well... he says, "No problem, come up and visit." We asked about that invitation to the sailors at Port Stanley Sailing Squadron and they all chuckled and said, "He WOULD say that, but you have to have local knowledge to get in there." They also gave me grave warnings about my 4' draft in Port Burwell. I have a feeling many of these warnings are overly cautious, but better safe and afloat.
You must watch the water levels as well. Although you can get away with a 6'+ draft in the lakes, it limits your destinations but also your time-frames. Later in the season, the water drops significantly and the silt fills in (dredging being done in the spring). So make sure that your routine destinations can handle the draft.
Charts are your friend... have a read through and look at places you can and can't get into. Keep in mind Lake Huron was only 0.2M (9 inches) above datum in July when it's supposed to be high-water time! You'll likely see that shallowing trend continue.
Having said that... 4-6' is pretty typical on the lakes, so not something you should worry about, just be conscious of.