My random thoughts...
The corner reflector (the spherical type with intersecting disks) is designed to work at all angles of heel since the idea of the "corner" is to create a 180 degree reflection using the two or three surfaces of the corner as long as the radar enters the corner +/-45 (90 degrees of returnable angle) degrees. In the spherical design once that corner rotates out of it's returnable path the next one rotates in so it works at all angles. The strongest return is when you have a whole disk aligned perpindicular with the source since you basically have two corners reflecting at that point, but on a bobbing rocking boat (The source is also rocking and bobbing) it's impossible to keep it aligned without some active alignment. The bigger the cross section reflector the bigger the return. Unfortunately the size of the disk that can be reasonably placed in a sailboat's rigging is somewhat small.
Putting metal in a metal mast won't accomplish anything. The radar is reflected by the mast itself, but the circular shape of the mast only returns a small portion to the source as the rest is scattered radially, and a boxed mast (square tubing) would only reflect, although very well, when it's aligned perpindicular to the source (vertically and rotationally). A wooden mast is basically transparent to radar so metal in a wooden mast would be of benefit.
Also, adding to it is the radar operator. If he has his gain turned down to eliminate sea clutter and only see the signature of a 500 foot steel hull so he doesn't have to stare at the screen except once in a while when there is a return, you won't be seen no matter what kind of dinky reflector you have.
AIS is cool but not everyone has them yet. And you can still run into another smaller pleasure boat not required to carry it.
So best I can offer is: Do what you can, keep a good watch when you can, keep some rum close at hand for the close shaves, and pray to the gods for safe passage.