Okay, I see where you're coming from now.
I think the issue here is the difference in perspective, with you coming from a powerboat background. In the proper PB, you can easily haul-butt out to the fishing grounds, do some fishing, then come back the same day. If unexpected foul weather is forecast, you can turn tail and get safely to port often before it arrives or at least before it builds to unmanageable. You don't even need a Hatteras or Post to do this, you can do it in a modestly equipped Bertram or even Grady White.
In a sailboat, you have to change your expectations radically. Heading out to the fishing grounds in the Gulf Stream, 50 miles off shore, will typically mean a 24 hour sail round-trip, and that's with no loiter time when you arrive at your fishing spot. With average sailing speeds in the 4-5 knot range, you will be committed to riding out whatever weather arrives while you are enroute. That means full safety gear, including EPIRB, liferaft, etc etc. $$$$$
Also, while many sailors do trail lines from their boats, sailboats as a general rule do not make the best fishing platforms. The cockpits are not designed for landing and subduing fish, and the abundant standing and running rigging makes handling fishing gear somewhat problematic. Typical production sailboats do not loiter well either -- they like to be moving along under sail for best dynamic stability in any kind of seaway. It can be tricky for someone inexperienced to handle the boat under sail at the slow and varying speeds and courses for trolling and bottom fishing.
Your budget will make it even more difficult to find a sailboat that suits your needs. At <$20K, you are looking at a fairly dated, small sailboat, that could handle the sorts of conditions you might encounter while out that far. With a small sailboat, my earlier prediction of 4-5 knot average speeds might be overly generous, given that you will likely need to beat at least one way to or from the fishing grounds. Probably best to expect 3.5-4.5 knots overall average, assuming a decent breeze.
So we were chatting and thought we could just take a sailboat out to the gulf stream and back for fun. Maybe after time letting a line off the back to see if we get lucky. Is this not something people do? and if not why?
Generally, this is not done. Primarily because it is expensive to outfit a coastal boat for this kind of sailing. Those who go to the expense to properly equip their boats will typically be voyaging further afield (like Bermuda and beyond), rather than turning around and coming back to homeport after reaching the 50 mile mark. Frankly, most sailors would view this as a largely pointless exercise (and somewhat boring!
), unless it was a shakedown trip in preparation for a more distant voyage.
Hopefully this info gives you a bit of perspective...