ichorniy - my slip is in the North Channel. I drive from Sudbury, which is about 1 hour and 15 minutes away, each weekend and sometimes after work to do a quick job. I don't think twice about it, the drive is great and goes by real quick. The sailing you'll get in Geogian Bay and in the North Channel is worth the drive, I wouldn't sail anywhere else at this time nor is anywhere else really available.
I do understand the grid lock you face in Toronto on fridays, maybe you can leave earlier in the day and drive back home later at night.
The problem you may find is getting a slip in Georgian Bay....but with so many power boats for sale up, slips may be more available. You're looking at Midland or Parry Sounds I gather?