How tight should I tighten the main halyard? I don't want to break things and over-tighten it...
The amount of tension you should put on the halliard depends on the wind strength. Generally, the stronger the wind, the more tension you should use, and conversely, the lighter the wind, the less tension. But, instead of thinking in terms of putting X amount of tension on the halliard in a windspeed of X miles per hour, you should learn to think in terms of sail shape
. The tension of the halliard affects the leading edge of the sail (the "luff). If you don't put enough tension on the halliard, the luff of the sail will be relaxed and wrinkled somewhat. Usually, you'll want to tension the halliard enough so that the luff will lie smooth and flat, with no wrinkles. If you tension it too much, the luff of the sail will have a long curl in it all along the mast.
Looking at your photos and your representation that they were taken when the wind was very light, I would have eased the tension on the halliard and also on the outhaul. Too much tension on the halliard and outhaul makes the sail flat, and that depowers the sails. When you are in light air, you want to maximize the power generated by the sails. Easing those two controls will give the sail a deeper draft, which is a more powerful shape, and will help drive the boat in light air.
By comparison, when the wind is blowing hard, the sails will generate more power than you need, or can use efficiently, so you should put more tension on the halliard and outhaul, to make the sail shape flatter, and reduce it's power.
The bottom line is that you should apply the amount of halliard tension that is right for the amount of wind on any given day.
The same principles apply generally to the amount of tension that you should use when raising the jib.
Within reason, you don't need to worry about damaging the sails by overtensioning them. Sails are very strongly made and reinforced, and they can tolerate a lot of tension, but they can be damaged if you grossly overtension them, so don't use all your strength and your body weight to tension them. Ordinarily, it shouldn't be necessary.
That sail is a North, full batten mainsail, and, when new, it was an excellent sail. It's hard to say, from photos, how bad it is now, but IMO it's probably still a lot better sail than many of the discount sails on the market. I'd suggest you not replace it for now. If you buy a good quality new sail, you'll begin to wear it out before you have learned how to make the best use of it. I'd suggest you learn the basics of sail trim and practice them on the old sail. There will come a time when you'll look at your sail and can see that it needs a better shape, but you can't achieve that shape through sail trimming techniques. When you are that knowledgeable about sail trim, that will be the time you should think about buying a new sail. I'll guess that you'll reach that stage after about a year of practice.