These Olympic class trapeze boats are really neat but they are not exactly casual daysailors. They really take big crew in the over 200 lb range to keep them on their feet and going. I have raced on them (Yes, I was and still am too light to be decent crew) and 505''s back when I was a teenager. Back then these were grand prix level boats.
They are very fast and fun to sail but are extremely challenging. They were fragile and had to be handled carefully. You can sometimes find one sitting semi-abandoned at a sailing club that specializes in one design fleets.
I do think that a boat like a Thistle, Lightning, or Highlander offer a reasonable level of performance and yet are more suitable for daysailing and messing about. If you are really interested in more speed, you might also consider a small high performance catamarran.