From limited experience as a guest on board in the Caribbean and Mexico:
Only the skipper goes ashore to clear customs. He/she must have ships' papers and passenger documentation in order. The cost of clearance and the logistics of doing so definitely differs from country to country. - maybe even from day to day and officer to officer in some cases.
The vessel should be under a yellow "Q" flag (and courtesy flag) until cleared in, after which the Q can be dropped. A vessel flying a Q flag in harbour may be boarded by local authorities if they are interested and well enough equipped to be patrolling. For that matter I suppose any visiting vessel may be boarded at any time depending, again, where you are. After doing the entire chain from St Maarten to Grenada we've never noticed any interest on the water from officials ( unless they were 'boat boys' undercover!
), though in Mexico the navy is far more obvious and active than in the smaller Caribbean nations who have little in the way of such assets.
However - visiting a foreign country/port without following proper clearing in procedures is to risk your entire situation unnecessarily. Your boat could well be seized and you'll at least be subject to a great deal more hassle and formalities and red tape (and cost) if you're caught than by following the rules. If no one sets foot ashore, it may be possible to anchor and move on without checking in... but again you should have the courtesy and Q flag flying.
It is equally important to properly 'clear out' and some countries won't let you in without proof that the last place you visited is aware that you've left. Occasionally this has required all passengers to attend the customs office in person and present your own documentation.
Flying in to join sailors is always interesting.. a standard question at the airport will be 'where are you staying?' and of course the majority will name some hotel or resort.. now you have to spell the boats name, indicate where it is, how long you'll be in country, etc etc..which can be interesting because we're not always aware of our hosts' plans....