That is a tough question. Of course people routinely leave their boats unattended at anchor and in most cases its perfectly alright. That said, I have seen some really awful situations develop. I had a guy come in with a 48 foot boat and achor very close up wind of my boat. (This was not even a crowded anchorage) I hailed the boat and warned them that they were anchoring too close to my anchor but they dismissed it. They had backed down to starboard of thier anchor and at that moment I was kited to port so we were pretty far apart when we were talking.

When they thought they were properly anchored, the two boats swung back together with then maybe a boat length (28 or so feet) forward of my bow. I hailed them again and they said they had a lot of scope ot and choked up on their rode.

They immediately went off in thier dinghy.

Then the boat started to drag. They ended up between my boat and my anhor rode between their keel and rudder and I could not even get my anchor up. I rigged fenders and tried to get my boat free until I eventually managed to get my rode below their rudder. They eventually dragged off behind me across the small channel (deeper water) and were probably a couple 100 feet astern (and aground on the other side of the channel) when the people came back for their boat.

That represents one extreme. If you carefully anchor noting bottom shape and depth and you are sure that your anchor is set well. If you spend a little time on board making sure that you are not dragging or swinging into your neighbors. If you are aware of wind changes and expected tide changes and keep an eye on your boat during these kind of anchor freeing events you should be OK going ashore.

Regards
Jeff