Chase, my oldest (now 8) was on board at 5 days old. Within 12 months of that, we were living aboard and cruising.
There really is not a bad age that we have found yet. Each one has its difficulties and its positives.
On the infant side, wherever you put them, they stay. The positives (all of this from a boaters/cruisers point of view) is that you do not have to worry about them wandering off. THey are happy and content wherever they are put. The negatives is that they cannot speak for themselves and you have to watch out for them geting too hot/cold. You have to make sure that during a tack they do not roll somewhere that they could get hurt or that something could fall on them. Also, they will not be able to swim. As such, you must keep a close eye that their location is safe. I will add another negative that we have found: it is hard to find any lifejacket that will work. I realize they make these infant lifejackets... but they are horribly uncomfortable. Both our kids hated them.
The biggest issue for us at this age really was the heat/cool aspect. They are especially suscptebile to getting too hot. Since you are considering cruising south, that will be especially true. On the nice breezy days, the cool ocean temps and breeze actually makes the "down below" quite comforable. But when the wind is not blowing and when it is in the 80's at night - it is very hard for everyone to sleep.
This was one of the principle reasons we went with the boat we did (a Catalina 380 at that time). We had to have a generator and a/c because we HATE being at the marina (contrary to what you read here!!!). We were always anchored out and the weather began to limit us. This was especially true in S Florida where during the summer there are rainstorms every day about 2:00 and you have to shut up the boat (and cook down below). So, I feel that if you are going to go cruising with kids under 5, you really will want a generator and a/c. We knew MANY people that did not have that and they will laugh at us for saying this. Many of the old timers that raised kids on their boats especially will laugh at me. But when it comes to the comfort and safety of my kids (and let's face it, a good night sleep for mom and dad), we felt that it was a worthwhile purchase.
When cruising with an infant, you also have to be conscious about forumla and baby food. The baby food comes in glass jars are horrendous when they break and a real liability on a boat. The forumla is not easily and readily available in anything but large grocery stores so you must direct your cruising and destinations as such (and plan on buying it and be happy if you can breast feed the whole time... Kris got mastitus (sp??) and could not). You also want good, clean water to mix with the formula and that is not always what you get out of your tanks. We used a Britta water fileter (and still do), but it is yet another pain to put up with.
Once they reach the two's-3's, they can be a bit more vocal about being uncomfortable and can push themselves up if they fall over in a tack (face down), etc. From that point of view, they are safer. However, they now can walk. They also do not listen any if at all. OPur biggest fear was always, ALWAYS, that Chase would sneak out of his bed in the middle of the night and go up the companionway. You will want to put a lock on it or something to keep them from being able to exit the boat at night. Luckily, boats in general are pretty child proof. We added a "crib side" that went across the V-berth for Chase that kept him very secure and in his bed. You can see a picture of that on the right side below. That is my mom holding him.
There was not a lot of other things that concerned us other than the electrical panel which is always open on almost every boat (with few exceptions). THere were many times he would get up there and play with the switches. Glen (my now 5 yo) has done that too. We were at the point of putting a plexi-glass cover over them before they finally started leaving them alone. However, many other people I know have rigged up something similar. You can see the panel here and how easy it is for kids to access it and start flipping switches.
At age 5-8, things really become a lot more "fun"... at least for us.
The kids enjoy swimming. Here is a pic of them swimming off the back of our boat and mom and dad's boat rafted up beside us.
They can take care of themselves. They can tell us when they are uncomfortable and now follow rules. The only "negative" of this age that we have found is that they also get boored and want a lot more interaction. You have to plan out your destinations as such. Long, all day sails are not fun for kids. Sometimes you do not have a choice, so you need to come up with board games, letting them steer, stand watch, etc. In other words, come up with things to help them spend their time and be a part of the boat. Still, they will get bored after a while and will want to go swimming or go out in the dink or something that does not involve boring old sailing (remember, my kids have done it since birth). Chase has really gotten into fishing and that is another great passtime to help the kids enjoy their sails or time at the boat. Hopefully they can even help feed the family!!! Here is Chase kissing his first ever "real" catch... nice one too!!
There are millions of other things to discuss. I know we really need to get that cruising with Children thread back on track. But at any rate, I hope this answers some of your questions and helps you out. I do not mind helping more.
PS Sailing, boating, and cruising are some of the best things we have ever done with our kids. They will mature faster than their peers, they will be vastly better readers, and they will learn a love and respect for nature. You will be close as a family. I have never met a cruising-kid that I did not like. It will be a life changing experience for you - more so than backpacking (which I did for all of my youth).