People have already mentioned the Caves at Norman island and the Baths, which are both great. One of my favorite spots is Soggy Dollar Bar at White Bay on Josh Van Dyke. A spectacular clear white beach and clear water where you can see the rodes
down to the anchor
while standing on the beach! There's also a reef snorkeling distance from the beach, and several really laid back real "beach bars".
It gets a little crowded on weekends with locals coming in, even from USVI, but worth the trip any time. You can take a $5 cab ride from Great Harbor, which is a better anchorage and famous as the home of Foxy's.
PS. I hope this is helpful. PM me if you want to to discuss more.