Port Jefferson is a great place to go to. I keep my boat on a mooring in Mt. Sinai harbor but spend just about every other weekend anchored up in Port Jefferson in different spots and also I have to drive through Port Jeff every time I go to the boat so I can let you know of tons of good restaurants and what not if you want to send me a PM.
Mt. Misery, aka Pirates Cove, is a real cool place. Its surrounded by large dunes on all sides, very well protected and is fun because you can easily swim or take the dink ashore and explore the dunes and get a great view of the Sound. Just like Jimscal said, def. wouldn't anchor in there but grab a mooring if you can. The only problem is on the weekends the place is jam packed with powerboats rafted 5 deep. Its tough to tell who's mooring you are on and forget about staying there Friday or Saturday night in the heat of the summer as you prob wont even find a spot. I would only go here on a quiet weeknight or spring or fall.
My all time favorite place to anchor is just across the main channel behind Old Field Beach on your way into Setauket Harbor and Conscience Bay. Once you get inside port Jeff Harbor you hang a right around the little sand peninsula into a smaller channel. On the south side of the channel you will see tons of boats on moorings and the north side is where everyone else anchors up. You can't miss it. Great holding ground - mostly mud. Water stays 10-20 feet deep right up to within 10 feet from the northern shore. On weekends it gets crowded but no where near Pirates Cove crowded. You can always find a nice spot to drop the hook plenty of distance away from others. Everytime I go I see tons of real nice sailboats from all over the Sound and even the East Coast. Absolutely gorgeous anchorage as you can see into Setauket harbor, beaches accessable only by boat and half of Port Jeff Harbor. Only downside to out there is that it is about a 15 minute dink ride into town. In my opinion it is def. worth it though. If you go on a nice night you will see what I mean.
The only other spot to anchor would be at the northern end of the main mooring area that runs just about the entire lenght of the harbor north to south and runs parrellel to the main channel. Just don't get too close to the giant 5000lb sand barge mooring. Downside to here is you will constantly feel the wakes of boats going in and out of the harbor with the ferry wake the worst which comes every 45 mins. It does put you a little closer to town though than pirates cove or Conscience bay does.
Mt. Sinai is another very pretty harbor but there are no real good anchorages or transient services unless you stay at the yacht club or one of the marine facilities like Ralph's Fishing Station. You could also grab a transient mooring. In my opinion Mt. Sinai is a awesome place to keep your boat - very pretty - great marine services - but is very quiet in terms of people staying on their boat and also with transients. I live on my boat every weekend and when I do stay on the mooring at Mt. Sinai there are very few others that are also on their boat. Where is with Port Jeff, regardless if you go to Pirates Cove or my fav spot near Setauket harbor and Conscience bay, every single boat has someone staying on it as these areas are just for transients and people who have moorings to keep their boat on for the weekend. No one really keeps their boats here permantly throughout the season so you get that vacation kinda atmosphere. Either way though you will have fun.
What's your boat named? I'll give a wave hello if I see you out this weekend.