If you were on Starboard and to Leeward of him, you would have ROW. Starboard has rights over Port and Leeward over Windward.
You have mixed two principles into one --
A boat on starboard always has priority over a boat on port tack.
The windward/leeward rule only comes into play for boats on the same