You can walk away and not do anything because you are afraid that the brokerage is going to spend money suing you, and let a whole bunch of less knowledgeable folk get screwed by a potentially unethical broker, or you can the right thing.
Unless you are talking about a boat worth several hundred thousand dollars, the broker is not going to sue you. And if you are buying a boat worth several hundred thousand, then the broker is not going to sue you either, because the negative consequences of you telling all your friends about it is going to negate any possible gain. There are not a lot of people who buy expensive boats out there.
Grow a pair and do the right thing. If the broker has a lawyer on retainer, all that gets him is advice and the first letter free. After that he is shelling out hundreds an hour to try to prove that you said bad things about him to a client.
If you contact the client and tell the client that you have a surveyor that you prefer to use, and that the broker refuses to work with that surveyor, you are not defaming the broker...you are stating facts. That does not consitute a tort anywhere.
If the broker still decides to pursue the issue, unless you spend considerable time and money telling the boat-buying public things about him that are not actual facts, then the most he can TRY to do is go after you for is his actual loss...which might be ten percent of the selling price, but also might be nothing.
He has to be able prove that he would have sold the boat to someone else either before, or very shortly after you spoke to the client, and that because you spoke to the client, and because you told the client things that were slanderous, or outright lies, he was unable to sell the boat to the other party.
So unless the guy has another offer waiting right now, and unless you do something other than explain the situation to the owner, you are in the clear.
About time that people stop cowering because someone might try beating them into submission with a large bankbook.