To go to the original questions. I wonder really how many places are restricted to a 4' draft. The French canals can take around 5'- 5'2". I understand the ICW is about the same, and in the Caribbean 6' draft will limit some of the places, but mostly you can anchor out a bit. Besides there is a wide choice of places to go. However, those are not my waters.
From a safety point of view, I don't know that strictly it makes too much difference but most offshore boats are 5' 6" to 6'. The greater depth gives a greater righting moment so less ballast is required lessening sail area for a given SA/Displ. That makes it easier to handle to a degree. Conversely the SA could be increased making it faster in the light.
The deeper draft could go to windward better which could be useful and a safety point but most cruising is downwind and longkeeled boats can go to windward if not as well.
The shallower draft models, say where it is an option, would tend to have more ballast to compensate for it being higher, and thus tend to be a bit slower, again SA/Displ.
Full keels are shallower slower in the light and make more leeway. Arguably they have a more comfortable motion, although there is some dissension on this. Probably a compromise of a cutaway forefoot is the middle ground.
Extra speed might have some benefit in less potential exposure to storms, but for a given size assuming trade winds the difference is probably under 5% which is 21 days not 20 on a very long passage. You could well more than make that up by reasonable sails and say a folding prop, and good trim.
Although there are areas of lighter airs this may be more of an issue in coastal sailing and downwind you have the option of a cruising spinnaker.
Offshore comfort and the ability to hold a course makes selfsteering easier and thus less tiring, which tends to favour the cutaway or full style.
I don't know the detail of centreboard styles but some qualify as ocean going. They would save leeway, be slightly faster through less drag, but because of less weight down deep have the disadvantages of shallower drafted boats in other respects. You could anchor closer but it is another thing to jam or go wrong.
One thing like draft cannnot be taken in isolation, because for a given righting moment displacement sail area, speed leeway and drag come into it. I think if you look at a designer like Perry or other successful designs you will find he uses mainly 5' 6" - 6' maybe down to 5'. This seems to represent the best compromise. Four foot is out of the ballpark, unless you go into cats or sail in a restricted area.
As someone pointed out most Americans can and do end up doing that.