First, layering is important, but you have to have the right layers.
The base layer should be synthetic, and something designed to wick perspiration and water away from your body. Most of the synthetic sports underwear out now will do this.
The next layer depends on how cold it is. On a nice day it might be pants and a shirt. On a colder day, it might be long underwear with pants and shirt over them. I'll often be wearing an REI t-shirt and set of Mountain Hardwear pants for my second layer, since they're synthetic, dry really quickly, and are tough as nails. BTW, the MH pants are fairly light, and can be converted into shorts... but, they're relatively warm compared to a lot of other pants I've seen.
The third layer is the real place you get the insulation. Polarfleece and related materials are excellent for this layer. Fleece provides a fair amount of warmth, even when wet, but is relatively light weight and doesn't restrict your movement much. When it is really cold, a set of heavy polarfleece sweats work wonders.
The outer layer should be either a good bib with a foul weather jacket or a drysuit, depending on how cold it is. A dry suit is better in colder, wetter conditions IMHO, but it is harder to regulate your temperature in a dry suit.
Head and neck:
Wear a fleece hat... 60% of your body heat is lost through your head. In really wet conditions, wearing a microfiber towel around your neck will help prevent water from going down the back of you foulies and soaking your clothes. Remember to wring the towel dry once in a while.
The high collar of off-shore foul weather gear is worth its weight in gold in really wet conditions.
Don't forget good socks... and boots that you can cinch the legs of your foulies around. Wet feet are no fun...and lead to blisters, and other problems.
The gloves have to be waterproof. I like 3mm Neoprene gloves. Ideally, they'll have a gauntlet type cuff that you can tighten your double cuff foul weather gear sleeves around—with one cuff inside and on outside of the glove.
BTW, insulating your torso is far more important than your legs most of the time.
If you can keep your feet, hands, and body core warm, the rest of it doesn't really seem to matter all that much.
A set of ski goggles does wonders when it is blowing like stink and driving rain/spray in to your face.
Foul weather gear:
The good stuff is worth the extra money you pay for it. It has to be waterproof... and the breathable gear is better than non-breathable gear. A high collar, adjustable length hood, neoprene, latex or PVC inner cuffs on the wrists, with an outer cuff on the sleeves, double flaps for the front zipper, drawstring for the waist and bottom of the jacket, velcro adjustable straps on the leg cuffs are all important.
Things to avoid:
Cotton... worst material you can have for clothing... it soaks up a lot of water and doesn't stay warm when wet. Wool or synthetics are much better choices for materials. Coastal foul weather gear seem to be water repellent, and in a really good storm, leave you soaked as badly as not wearing anything at all.
Shivering is always a bad sign... it means you're losing too much heat, and that your body is trying to generate more by shivering. If you're shivering and then stop and get sleepy... hypothermia is setting in... and that's really not good.
Keep well hydrated and fed. Your body needs fuel on crappy days. Avoid alcohol and caffeine since both tend to dehydrate you and cause you to loose more body heat more quickly. Alcohol is worse than caffeine, so if you need one or the other, go for the caffeine.