I have admired that boat for years, but have never had the chance to take a close look at the rudder. From the diagram below my guess is that the only chance you have for not removing the rudder is if WEB Crealock designed the rudder with a hole in it for shaft removal. On some boats there's a tube built into the rudder located so that when the rudder is hard over one direction or the other the shaft will come out through the tube. In normal operating mode, the tube is covered / faired with a filler of some sort so it doesn't disrupt the flow over the rudder.
Is the boat out of the water now, so you could look at the rudder?