Maybe a little more info on your shooting skills is in order. Shooting off sand bags if you don't know how isn't very effective. This is what I do to a new rifle or old one for load development, to "check it out".
50 to 100 rounds should knock the bark off the bore and smooth it up enough to start shooting half way decent for FACTORY barrels. Polishing with 200 strokes of Flitz and a fairly tight fitting cleaning patch on a jag before firing any rounds also works well. I usually use up the odd amounts of powder and bullets to break in a factory barrel along with polishing with Flitz combined to equal 50 to 75 rounds. You can feel the bore smoothing up and I quit when it gets "just right".
Take it to a 'smith if you don't feel comfortable working on guns.
First, clean the barrel well with any of the newer cleaners. I now use or have used just about every cleaner the market has produced including Wipe out, Butches Bore Bright, Tm, Shooters Choice, Montana Extreme, Flitz, Iosso, J-B and new one called KG12 I found on the 6mm Benchrest site which is so good I can't hardly believe it. Clean down to bare metal, then shoot a few to see how many you need to foul the bore evenly. Usually 2 to 3 is sufficient, then fire that many before you start shooting for group. An extremely clean or dirty bore won't shoot the best.
Tighten all the screws: Scope base, rings and receiver, and make sure the scope is good. Swapping in another scope sometimes speaks volumes.
Run a business card down the barrel channel. If it touches mark the spots so you can clean up the channel. The cards are about 0.012" thick. I like about 0.030" on my shooters.
Are you using factory or reloads? If reloads, are you building them and are you experienced or novice. Bad reloads can make for patterns not groups. If factory, try another brand. You need to be more specific about the group size. Saying every brand shoots 4" groups doesn't convey much information either. What brand shoots what size with what bullet at what velocity and where on the target. MEASURE the groups and write the information on the targer. Very few 150 or 180 bullet shoot to the same point of impact.
Have the trigger tuned or replace with an aftermarket. I like Jard because they are cheap and work, but I also have Canjar, Shilen, factory Rem 2 oz, Timney etc. I set mine to 1 1/2 lbs so all my rifles are consistent. No guessing.
The recoil on the 300 WSM isn't hardly enough to worry about, just slightly more than an '06, except if you believe the hype or weigh 100 lbs soaking wet. If recoil bothers you get a nice Limbsaver recoil pad installed. I love them and almost all my rifles wear one. Recoil is more in the mind than in the rifle, arguably, but a nice soft pad works wonders in that department.
I use a 20 or 36 power scope for load work up on any of my rifles when I'm working up loads and a 2 or 3" target dot. You need to see what your are shooting at to get the best results. I can't even see a 1" dot at 4 power anymore at 100 yds. Even my 45-70 and 416 Taylor get the Leupold 6-20 or Weaver 36 when developing loads.
Keep your eyes open, both of them, and learn trigger control. Spend 15 to 20 minutes each night dry firing to see just how much you jerk around. The sight shouldn't move off the target when the firing pin goes click. If it moves, where it stops is where the bullet would have impacted on the target. And, learn breath and heart beat control. You can't shoot straight if you are puffing and panting and your heartbeat is banging the rifle around.
My recommendation, unless you can 'smith yourself, is to shoot a few 3 round targets with the bullets you want to use, then take the rifle to a competent 'smith with the targets and seperated brass also. If he isn't interested in the brass or targets, find another 'smith. Both will tell a very interesting tale. But be careful that he isn't just trying to sell you a new barrel job. Get several opinions.
I would guess the barrel is OK and solution lies else where, but you never know until you work through the many solutions. Don't let an "expert" pump you full of smoke. Blaming a barrel is usually the first thing pointed at and most of the time the last actual problem.
I would suspect doing the barrel channel clean up, trigger job, recoil pad and maybe the scope change, will solve the problem.
Finding the right bullet, powder, primer and seating combination would be the second part of the accurizing process. Keep diligent ACCURATE written records, no BS or ego stroking. Now days factory premium ammo and a factory rifle should shoot into 1 1/2" right out of the box, but that is not always the case. Read as many articles as you can find on 300 WSM tests to see what the "pros" came up with, but remember they work for the factories indirectly.
I used to accurize rifles for free, just for the information and experience and because I kept the unused components or any tools etc I needed to do the job, well over 200 over the years and if I couldn't reduce the groups by half of more, it truely was a lemon and a new barrel was the only recourse. I can remember only about a half dozen that went the new barrel route. My goal, depending on the caliber, is 1/2" for up to 6mm, 3/4" for up to 338 cal, and 1" for anything above 35 cal. I've been able to hit those numbers or slightly smaller almost 90% of the time. It takes patients and know how and keeping the ego and temper in the freezer. You can't do well if you are trying to prove something or get pi**ed and throw tantrums when things don't go right. I could tell you stories...but I won't.
Lastly...don't jump around trying this and that. Set up a plan of action, think of the rifle as a system with several component groups. Scope and mounting, barrel, trigger group, bolt and action, stock, etc. Start with the scope and mounting, fire a few groups, do the barrel channel, fire a few groups, do the trigger, fire a few groups, lap the bolt, fire a few groups, all the time writing good notes, keeping your targets and practicing good breath and trigger control and reloading techniques. Besides learning something about your rifle all the repetition will teach your muscles something also.
Of course all this good information means **** if you're not up to doing it or just want to play on the 'net...
Nothing I stated is ment to offend, just my way of talking straight. If it fries your tater, well.....
I've been doing this thing for half a century and learned a few things. I try to pass that information on, but you know the story about the horse and water.
Let us know more information if you are serious.