I think the right thing is to fly the US flag, not the Yacht Ensign 2/3 of the way up the leach of the aftmost sail unless you are gaff rigged then host to the peak of the gaff when sailing during daylight hours. If under power then host the US flag on a staff at the stern during daylight hours. And you should show your colors while entering a foreign harbor at all hours of the day at the proper pace along with the ensign of the host country at the starboard spreader.
The Yacht Ensign is really a signal and was used to indicate a US yacht not needing the service of customs when entering a harbor. Of course the use has changed over the years but you should stick to the US flag if you are sailing outside the US and use the Yacht Ensign in home waters if you like it.
All the best,