I used to sail on a small inland lake, and could easily swim a half mile or so to shore if I fell off the boat, even without a pfd. On that lake, I rarely felt a need to wear a pfd, and never wore a harness. But, now I'm doing a lot of singlehanding on the Chesapeake, and have jacklines, harness and tether. I use them whenever the conditions freshen up, because when you get away from Annapolis, there are a lot of areas where you might not see another boat for a long distance, and you can be a lot more than a mile from shore.
I'm not afraid of risk, having engaged in lots of fairly risky activities in my lifetime, but, regardless of how much risk is inherent in sailing the Bay, a harness and tether reduces it.
Years ago, I was about a mile from a sailboat when it rolled over while entering a pass, resulting in the loss of a life. That life wouldn't have been lost if the person had been wearing a harness and tether. That one object lesson was enough for me to realize the value of a harness and tether.