I had an experience with my current boat where I knew the broker (bought two other boats from him) and their was a competent surveyor that they bared from the marina. This surveyor was also commonly employed by an major insurance company, so his "surveyor credentials" were not in question.
There was an offer on my present boat and the potential buyer forced the yard to let this surveyor perform the survey. When all was said and done, the surveyor did his "normal" thing and supposedly found many items that eventually killed the deal. When I made my offer I had a surveyor who was familiar with the boat and also the "other" surveyor. My broker shared with us the outcome of the previous inspection and the issues. My surveyor paid particular attention to them while he completed his survey. His familiarity with the boat model however allowed him to explain to me why the other surveyor was incorrect in his concerns. In the final analysis the boat was pronounced sound, I purchased it and have been happy for the past 5 years.
My surveyor hinted that the surveyor in question killed allot of very good deals through out the area. Some because he thought his job was to find as many things wrong with the boat as to impress his customer and offer areas of renegotiation (which is a half truism). Some areas he would voice concerns saying a "possible major problems" with little or no experience to back up his promise of doom. Further investigation seemed to prove that this surveyor was never happy with a boat over a certain age and under a certain price, therefore he was bared from the marina, and as my broker latter confesses it was to save the customer the cost of a survey that they new no matter the condition of the boat, the buyer would end up with more doubts then were necessary. My broker even offered to pay for a second survey for the potential buyer of my boat to prove the boats worthiness but by then there were so many questions the deal die. Thank goodness.
My advise is, if you trust the broker find out his concerns.. if you don't and you really want the boat your surveying plan on a second survey. therefore if the first does have issues you can confirm them.
Like others have said if you don't trust someone in the deal... probably better to walk away.
I think your broker was at least honest to say out right he would not deal with this surveyor. He could have just made it inconvenient and tried to steer you to another.