50-70,
Have you checked the twist? If not, mark a cleaning rod for length and angle when you've just engaged the rifling forward of the chamber, push a tight patch through either until you've made a full revolution or a half revolution, and measure the length it takes to do that. If you can only get a half revolution (I think that's what will happen), multiply the distance by 2 to get the twist rate. I dn't have a good reference for RB twist rate, but the 50's were typically in the 36-40 rate, which limited them to the 450 grain bullets.
If you really want to be competitive in silhouette, you probably need at least a 600 grain bullet in 50 caliber to be efficient enough ballistically for low wind drift at turkeys and rams. More than likely, your rifle won't stabilize that long a bullet. If it did, you probably wouldn't want to shoot it prone because of the recoil in a relatively light rifle.
Your rifle might work for Scheutzen. but most of the rifles used for that game are capable of 3-4 inch groups at 200 yds. to be competitive for the bench portion. Your rifle might be that good-I have a 43 Spanish RB that I believe is capable of consistent 2 inch groups at 100 yards.
It sounds like you have a great, fun plinking and hunting rifle. Or, you could always rebarrel with a heavier barrel and turn it into a very competitive silhouette rifle like many others have done.
As to the primers backing out, measure the depth of the rim recess and compare it with the rim thickness. If you have much more than 0.005" clearance, you can get that happening. The firing pin pushes the cartridge forward, the pressure holds it forward, but there is nothing to restrain the primer from backing out. Often, this will also give you short case life, with case separations. The answer is to get thicker rims that fit your chamber. Buffalo Arms does this for other cases by swaging the rims thicker, and it works well (they take .43 rims from 0.060" to 0.088" or so) and solves the problem. The thin rims in the .43, incidentally, causes some separations after 2 firings.
Clarence