The downfall is that the listing broker doesn't represent you and never will - they have a contract with the seller to represent the seller. Don't confuse yourself about the listing broker's loyalties.
You certainly can deal directly with listing brokers, you owe Joe nothing. Just make your own phone calls and do your own footwork. However, you arn;t likely to save any money by not having joe in the loop, if he participates in the sale the listing broker shares his fee with Joe.
If Joe is good at what he does he can save you time, effort and money by pre-qualifying boat types and specific candidates so you don't spend time tire-kicking boats that won't fit your needs or expectations. Even if you use Joe, remember he's being paid by the seller, so take any advice with a grain of salt...I personnaly think working through one broker you trust (with care) is the way to go.