They should move either way. You should be able to drive them with a brass drift. If not, put a screw in and screw it down against the barrel under the dovetail insert; not a lot, but some. This should loosen it a bit so you can move it. If you overdo it, you'll have a loose insert, so be careful.
RonF