I agree with TakeFive's approach. I wear a harness and/or a PFD, and I like the flexibility of being able to wear a harness without a PFD in extremely warm weather or a PFD without a harness in certain conditions, or both when necessary.
I have the cheapo Jim Buoy variety with the supplied rope tether. Here is a photo of me off Delmarva during last summer's heat wave, with my Petzl red headlight for reading charts: https://plus.google.com/photos/10193...Pq64djL942rzQE
Incidentally, you can see my aqua colored beanbag to the left of the photo, for sleeping in the cockpit. High temps were in the 90s during the day with high humidity and lows in the 70s or 80s at night.
My harness clips to a dynamic (stretchy) climbing rope that I use as a jackline, to lessen the impact stress on all points. In the unfortunate event I fall overboard, I want to be able to fall clear of the boat relatively gently, slide quickly aft behind the stern, and climb aboard. Hopefully, I will never test the system in real life.