A few things to think about.
Digital scales can have a variance of up to 3 grains in some models and is stated in the fine print that comes with the scale. I have 3 different digital scales and each will weigh the same bullet differently. Try comparing to a balance beam scale...and those even have a few tenths variance unless constantly adjusted.
4 grains isn't much in a 400 grain bullet. 1/100th, 0.010th of the difference. Unless your gun is perfect in all respects and will shoots into one hole and you can hold it that well, you're comparing Mt. Everest to a pimple on a flea.
Take the heavy bullet and shave off the 4 grains and see the actual amount...have a magnifying glass handy.
Besides...think about what you do to size and lube those bullets...even if you balance an as cast bullet to a precise weight as soon as you run it through the lubrisizer you just messed all your work up and have to start again...not to mention the fact that as soon as you touch off the round the flame will burn off a bit of weight, roughness in the barrel will scrape of a bit, the lub will come off and there is no way you can determine the weight of the bullet as it exits the muzzle...and each bullet will loose a different amount.
I'm always surprised just how accurate cast lead OR jacketed bullets really are in today's world.
If you want to be consistent...do what Lone Star says...weigh separate your cast bullet into weight groups and shoot 5 round out of each group to see if you can find any difference in the groups. Unless you weigh each powder charge to less then 0.001 gr, weigh your brass, primers, point the ogive touches the lands, lock the rifle down so it doesn't move at all and all the rest of it, it would be hard to distinguish a 10 gr difference in some rifles.
Benchresters that have almost perfect rifles and do unimaginable things to keep consistency have a hard time getting perfect bullets even with bullets that are made as perfect as possible.
Cast away and don't get too persnickety...it will make you crazy. Enjoy your rifle and your loading and keep things in perspective. Weigh separating your components and being consistent can lead to better accuracy, but the consistency part is the most important...consistency in all aspects of how you reload will produce much more accurate ammo.