If you want to hear weaker and/or more distant transmissions you should turn your Squelch down, not up. Generally you should set the squelch to the lowest possible setting where you don't just hear white noise/static.
It's instructive here to have the base unit VHF and the handheld monitoring the same channel. The 40 foot or so higher antenna of the base unit (and the likely greater sensitivity) when compared to the handheld with a touch of squelch means you miss a fair bit of traffic merely by using the handheld in the cockpit instead of a remote speaker.
I performed a test once where I went about 6 NM downrange on a Zodiac with an intervening spit of low land between myself and my wife on our main boat with the 40 foot of height. We used a pre-arranged time and channel, and while I could hear her (maybe a 3 out of 5), she missed about 40% of my replies. By contrast, on the handheld and on open water, I am almost always able to reach the CG to report logs, picnic tables and assorted other nav hazards in the water. The same CG cannot frequently hear the handheld (a SH 5 watt "typical" VHF) from the club basin, probably due to nearby buildings and the forest of radio-reflective masts.
This isn't necessarily a problem, of course. 95% of VHF traffic on 16/9 is trivial and rarely needs one's full attention.
But it can help one set the handheld squelch effectively: A perfectly received but distant (say, more than 4-5 NMs) call on the base unit will sound like static with a few faint words littered here and there on the handheld. This is why I usually monitor 16 on the base unit and use the handheld for weather alerts, etc. or for calling other boats within visual range. If I hear a pan-pan or something, I will switch the handheld on deck to 16, but I don't expect to be heard at more than 5 NM of distance by other boats.