From your story, this is pretty clear;
So i was sailing down wind with little to no wind doing 1mph. motor was off and up just using the main sail. so there was this fishing boat about 40-60 ft local lobster boat. i was staying same course and he was near the shore then he started coming out to the middle of the bay were i was and i thought ok he'll pass port to port but then as he got closer he changed course or something happened it looked like he was coming at me now.
You were a sailboat - with atrocious grammar. It should have been clear to any observer that your motor was off, and not in the water, so you were not under power. He was a Lobster Boat - i.e. NOT a vessel constrained by draft, nor his rig. You were the stand on vessel, and therefore obligated to maintain course and speed, unless a collision was immenent.
A call to the coast guard reporting this clown would be a service to all other boaters in your area, but it will probably also get you shot. Think of it as taking one for all other boaters.