I do not wear one all the time on our boat. While these could become my famous last words, it would be pretty tough to fall out of our cockpit, through the bimini frame then through the lifelines/pushpit. (throws salt over shoulder)
However, I will wear one in rough conditions or on lengthy passages, where it is just the two of us and one is sleeping or down below.
On a boat which would be more likely to knock down or easier to end up in the drink, I would also wear one more often. If one is alone, I see no excuse not to wear one at all times, with a handheld radio, regardless of the boat. Seriously, what would the back up plan be, otherwise?
As for the feedback that inevitable proclaims that some wear them all the time, no matter what, etc, etc. My observations of sailors over 40 years, is they must sail well out of sight distance. I see very few come and go with their pfds on, all the time. I find it somewhat amusing that the entire crew in the Newport-Bermuda race is required to have them on when the cross the start and finish lines, but fielders choice in between. They are trying to set a good example.
While it remains good advice to wear it at all times, there are some that claim to, but have posted several pics of themselves sans pfd. You know who you are.