It is my understanding that the 30-06 was designed to shoot 165 gr bullets. If you shoot 165 gr boattails, you should be doing what the designers hoped for.
Having said that, you did not state what the twist is in your barrel. Knowing the twist can tell you approximately what weight bullet you shoud be firing. Oddly enough, my 06 doesnt really like 165 gr boattails, it likes 150 gr hunting loads alot better. Most boattails do not have a good performance after the hit.
Even the component parts of the ball will affect accuracy. Not all bullets are created equal, and accuracy pays. What you have so far is just about all anyone can hope for out of a modern mass produced rifle. You can tinker if you want, but I doubt it will get tighter.