I haven't experienced a COB but I have practiced it with several crew. As with anything, practice is so important. Practice makes response automatic so that you can think ahead and handle the unexpected while executing the practiced plan.
I did COB drills on a 50 foot bareboat with 8 on board in the Grenadines. The experience level was much lower than my wife and I who've had ten years part time sailing and a few courses. I spent quite a while before we left the dock just organizing my notes. Then I did a very serious minded presentation in the cockpit in the slip the morning of departure. When everyone was relaxing after the presentation I called a surprise drill and they responded fairly well. It took two drills later underway before we actually got it right and it was quite educational for me also.
Something I had to handle as skipper were those spontaneous ideas which are well intentioned but not necessarily helpful. I anticipated that with a simple statement in the briefing defining the role of the skipper, the need for information and the way to communicate it, and the proper time for discussion.
I trained everyone to at least understand all roles, and there were two people for every designated role. Of course I had to have a backup skipper too. The next time I do such a drill I will sit it out and have the backup take over.
For the underway drills I secretly made a dummy crew member target with three mostly full one gallon water jugs and one empty inverted one lashed together. I put a piece of loose leaf paper in the empty one to give it some resemblance to just a human head in the water. When it was drill time I quietly tossed the thing from a transom locker and waited a few seconds before calling the drill so they would have to find the target too. I've heard that the US Navy sometimes uses a coconut as the COB target.
Included in the training was the role of the COB so everyone knew to hold their biggest piece of clothing overhead regardless of modesty if they ever were overboard.
After that trip I decided that for any future bareboats I will be incorporating the COB practice early, including pre-cruise materials. If people won't commit to that up front then they simply won't be on the trip.