What two boats sailing within sight of each other are not racing?
In my opinion, I guess it depends on how competitive you are if you will be happy with the compromise. I am very competitive and like to go fast, but as long as there are other boats in a class that I could sail against head-to-head, I can be satisfied. I would get bored pretty quickly if I was only able to compete on corrected time.
Speed is relative and you don't have to be on a sport boat to feel the adrenaline rush from a close contest.