I concur with Valdare - for men a collared shirt is almost considered "dressed up"; that includes places like the Bitter End and Peter Island. Much more than that is tough on the apparell since you'll almost always be coming in via
dinghy. I like a pair of those ultra light long pants since they perform the functions of (a) insect barrier at dusk/dawn, (b) "long pants that are not jeans" for discerning establishments and (c) SPF coverup for sunburn protection or coverup.