It did not think that those light weight bullets are varmint bullets myself but more along the line of a Single Shot pistol type.
35 is a nice caliber, the only one that was ever chambered in the Handi is or was the 35 Whelen. It can be handloaded way down using H-4895. With a 180 gr bullet you can start with as little as 36 grains of that powder which will come very close to 35 Rem levels. As you princess

grows, the performance can be increased.
Since you brought up you had a 30-06 Auto banger I wonder if you get a chance to load or have someone you trust make some mild stuff just how low you could go for the action to cycle.
I have a 742 that cycles with 44.5 gr IMR 4895 pushing a 150 gr bullet. That is the minimum powder charge from a Sierra book for that powder. The semi action soaks up a lot of felt recoil.
Regardless of what you pick, the stock has to fit her that is more then half the battle.