My impression is that the Firsts are not just much faster but better built. But I have no experience with the Oceanis, just the Firsts.
I don't think the biggest difference is the building quality but the different design criteria and most of all the sailing hardware (winches size, fittings and deck equipment) that is of remarkably better quality on a First.
That is what makes a First substantially more expensive than a same sized Oceanis, never mind the bigger and brighter interior of the Oceanis, that probably cost more to produce.
However if we are talking about the First 40, the current model I would say that the boat has a "small" detail that makes it unsuitable for cruising: I am talking about the ridiculous space of the chain locker.
Contrary to the previous model, the 40.7, the First 40 is not popular among cruisers due to that handicap but while you wait for them to sort it out (it has taken already a long time) you have boats with the same sailing criteria that make very good cruisers like the Dufour 40, the Salona 41, the Dehler 41, just to mention some.