I have Handis in both .308 and .30-06.  I shoot and hunt with the .308 more than the '06, and it has a steady diet of 165 - 168 gr. boat tails.  I don't mind the recoil at all, and it is on a synthetic stock.  The '06 on a wood stock seems to kick more with a comparable load.  Of the two, I prefer the .308.  I've had a bolt action .308 for years and really like the caliber and what it can do.  When I got the Handi, it just about retired the bolt gun.  If you want some long range clout and light recoil, you may consider the 7mm-08.
Pete