Well, to me, the York is a prettier river. But if it's plantations you are after, then by all means check out the James. The James is much wider, but still has a fairly narrow navigable channel with a lot of tug and barge traffic to watch out for. You will have to be careful approaching close to shore in many areas due to shoals and fish traps. Don't let me scare you off of it though, there is a lot to see.
We spent last Labor Day weekend anchored off of Chippokes State Park, which is located in the bend just opposite Jamestown. It's an old plantation that has been turned into a park. Anchoring is free, and you can use the campground showers. They do tours of the old plantation house and that sort of thing.
Smithfield is also nice. It is located up the Pagan River, which branches off of the James just past the James River Bridge. The Pagan has a narrow twisty channel with a lot of current on the tide changes, so it provides a little adventure. The old town is good for a day of walking and sightseeing. The do a lot of music festivals and stuff there too, so check the events calendar for the days you'll be in the area to see if anything looks good to you. You can anchor in the river off of the marina and use the marina dinghy dock and showers for a small fee if you don't want to stay in the marina proper. Just be aware of the current, it really rips through there on the tide changes.
For the high rollers, you can stay in the marina at Kingsmill in Williamsburg. We've never stayed there, but have been told that it is very nice with access to all the amenities at Kingsmill plus hourly buses running to Busch Gardens.
Although much diminished from its heyday, the Ghost Fleet is still there in the James and is fun to sail past.
Have fun and let me know if you have any specific questions for me. Happy to help. I almost never find anybody on the forums coming down to the lower bay unless they are passing through Norfolk to the ICW, so it's good to have somebody coming down just to check it out.