We reduced the liklihood of sea sickness by sleeping on the boat overnight before heading out, the sleep allows the inner ear to relax and adapt to the boat motion.
Others mentioned keeping busy, my sons managed the water providing drinks when they thought we were thirsty (otherwise known as water play) other things could be getting a squirt gun and allowing them to spray as much as they like (salt water is cheap entertainment).
As the kids grew older we would put them leg over on a staunchion and oversheet on a reach so they could splash in the water. Did this once through a racing fleet and had the crews laughing at the idiot with the boat oversheeted then they saw the kids and had a greater laugh when they realised the entertainment. Never lost a kid but you do need to look out for the big waves.