We own a 22,000lb boat (plus water/fuel/people/stuff) and we'll be double reefed on the main and single reefed on the genny at 25kts. We'd rather be comfortable and slow than hard over and a bit faster. Actually, it depends on the time of day. In the morning, when we first leave, we don't want to be hard over. By the end of the day, we really don't care and just want our first drink
To answer the original poster's question, I think you were a bit overcautious by saying that you had to lower sails and get all hands on deck, but that's much better than undercautious (although a sailboat will handle better in almost all conditions with some mainsail up). On our boat, the kids wear life jackets when not below, but they can take them off below. For the grown ups, life jackets go on either when we put the first reef in or the seas are high enough (or confused enough). If the wind blows over 20 or in inclement weather, we usually string our jacklines out and clip in. When there's a lot of stuff going on (reefing, frequent tacks), if our kids don't want to be involved we have them go below. As others have mentioned, it's critical that you have a safe environment for them to hang out in. If it doesn't exist today, build it.
I think your skipper was a bit harsh on you, but it's a difference in experience. You're more nervous at higher winds whereas I'm assuming he has been in more and knows how the boat will act. On our old boat, I was great. On our new-to-us boat, I was nervous the first few times the wind blew over 20, and I'm still getting used to it as I learn what she will do and how long it will take for her to dig in.
Just persevere, and get more experience at the higher wind velocities. Put reefs in BEFORE 25kts of wind, learn how your traveler can reduce heel and weather helm, and learn how your boat will react to gusts.