I could be wrong, but I swear I read somewhere that the bridges over the Swinomish were closed for emergency maintenance right now. Not sure when that started or how long it was supposed to last, but I would check before finalizing your plans... you might not have the option of going up the inside at all.
Fastest I have ever done it is also about nine hours, catching the tides just right. I haven't made that particular trip often enough to give an average, but I go between Shilshole and Port Townsend regularly, and the fastest time has been around five hours and the slowest two days. Wind and weather can cause your trip time to run the gamut, particularly once you hit Admiralty Inlet (nine times out of ten we duck through the Hadlock Canal and avoid that entirely, but it will be further out of the way for you).