The advice given has been great... layering, especially with polarfleece is a good idea. Fleece is great stuff, since, like wool, it is warm even when wet. Old Navy has good fleece stuff, but they probably don't have it in stock any more... it's rather heavy though. Sportif makes some lighter weight fleece.
Avoid cotton clothes or underwear, since if it gets wet, it will wick the heat from you and leave you seriously chilled. Wool and the high-tech synthetics are good. REI and many sporting goods stores have the appropriate layers... base layer should be wicking materials, then insulation layers—depending on temperature, and finally a wind-proof/waterproof shell.
Breathable outer shells are better than non-breathable. If the foul weather gear you get isn't breathable, make sure it has some good ventilation. Get the foul weather gear in red, yellow or orange... the more common colors of white, blue, green, and black are really tough to spot if you fall in.
If you want to get "marine" foul weather gear, I'd recommend the West Marine Third Reef
line, which is pretty affordable and works pretty well. The other brands, are a bit overkill until you find out whether you'll continue sailing.
Good gloves are essential. 2-3mm Neoprene gloves are really good, and keep your hands pretty warm and most are completely waterproof. Jackets should have sleeve cuffs that you can cinch down to prevent water from dripping in through the cuffs... neoprene, latex or PVC lined cuffs are better than plain nylon or polyester cuffs, as they seal better. The marine specific gear often has inner and outer cuffs, to better prevent water ingress.
If you like wool... get merino wool, as it isn't scratchy like most other wool, and relatively affordable.
Get a fleece hat for your head... you lose a majority of the heat through your head... The one I have is from UnderArmor
Deck shoes... get the ones that look like sneakers.. they're better and more comfortable than the ones that are leather... also, I think they last longer. Harken makes some good ones... If it is going to be colder, get a pair of neoprene dinghy boots
... they're waterproof and will keep your feet warm and dry. For socks... get merino wool socks, like the ones made by SmartWoool.
Sleeping gear... either a synthetic sleeping bag or fleece blankets... again—because they'll keep you warm even when wet. Bed Bath & Beyond sells fleece throws that are pretty nice, and I keep two or three of them on my boat for guests, emergencies, etc. If you're really tall, they won't work.
Get the foul weather gear oversized, and try it on... make sure you can put a few layers on under it...