I did essentially the same thing when I put a tube between the holes for the tabernacle hinge
pin on my mast. The tube keeps the pin/bolt from wandering off target when inserted from one side. I used a piece of aluminum pipe, and instead of cutting the ends flush with the mast I made the pipe 1/16"-1/8" longer on each side. I used a ball peen hammer to peen over the ends like a rivet and then filed everything smooth. It works fine and hasn't budged in 15 years.