I haven't found the .25-06 to be any louder than, say, a .270, or most other similar cartridges. Still, I always wear hearing protection.
As for the Barnes X, I love it. Despite the fact that all of my guns which are loaded with the X are very accurate, there is no denying that in some guns they group horribly and, thus, people hate them.
It's nice to hear that they shoot well in your .25-06. Given that they retain 100% of their weight, I think that a 100 grain X would be perfect for deer sized game in the 25-06.
Zachary