It is aggravating to look at (so I dont much look anymore)

......
I wouldnt send an otherwise good fit and good shooting barrel back for something like that; just dont want to take the chance.....
If relieving the screw hole doesnt help, or sanding only a bit, then something is off, likely the lug, but what bugs you is the spacer sits off. So, remove it, fill the holes with glued wood plugs and do a careful center-up, hole spot and drill it to fit the frame where you want it and screw it back on. Pay attention to how the wood meets the spacer though, but it usually doesnt take much and it wont bug you anymore.