If in fact you were on starboard and they were on port, you had right of way, and should not have had to alter course to avoid him. If he forced you to deviate from your intended course he has fouled you.
If he had gybed cleanly in front of you, and became leeward boat on the same tack/gybe, the onus falls to you to keep clear.
However, if you then became overlapped approaching the finish, he cannot "stuff you up" and force you on the wrong side of the finish
line (like he could to a barging starting boat).
As I write this it occurs to me that the 2 boat length circle may apply to the establishment of the overlap like it does at any other mark.
Fun, huh?