"Two ships pass in the night, no one notices. Two ships pass in the day, it's a race."
Absolutly, you can do both. It really is a matter of degrees. Yes some boats are set-up to for one option or the other, but most of us have boats that were specificlly designed to ride the fence between the two modes of sailing.
Sometimes you have to make minor modifications to your cruising boat to race. There is a guy in our harbor that keeps his boat on a mooring. He occationally likes to join in the Saturday morning races. I regularly see him row out to his mooring and then off load 200 ft. of chain rode into his dingy before the start. Me, I keep a cardboard mock-up of a BBQ on the stern rail so I look like a cruiser, but without the weigt.