I don't have much knowledge of the East River, but there are 3 marinas that are convenient on the Jersey side of the Hudson:
Newport Marina is a block from the PATH (subway) into Manhattan, has floating docks and is rather affected by the river.
Liberty Harbor is convenient to the PATH (5 blocks), out of the river, but has a noisy bar on Friday / Saturday nights. It also has a ferry leaving from the marina.
Liberty Landing is on the south side of the Morris Canal, across from Liberty Harbor. It also has a ferry and is inconvenient from the PATH.
If you can't get into 79th St or have some reason not to, they are reasonable choices.