My .338 A-Bolt does the same thing except in the opposite direction. The first shot out of a clean cold barrel is about 2" higher and 1" right of where the rest hit. Sometimes it's the first two shots. I'm convinced that it is the clean barrel that is doing this. I have shot it twice, (hitting high right) then put the gun away. Came back the next day and all the other rounds land exactly where they should. I have done it enough times to prove (at least to myself) that it's the clean barrel.
Let me ask you this,, Has the barrel in question been Fire Lapped?