JVS: Your relationship with your grandfather mirrors my own relationship with my hunting elders way back when I started deer hunting at 16 years of age. I hung on their every word, and took their words as gospel as these were the great white hunters who convinced my that my new Savage 99F in .308 could not possibly have much reach and that they could pull off shots with their .270s that I should not attempt. I lost confidence in the 99 and sold it for a bolt action 06 with a 24 inch barrel that was a real pain in the butt to carry through the eastern woods. But, the rifle met with their approval, making me feel more like one of the guys. Now that I've got 50 deer seasons of experience, I have never seen or taken a shot that could not have been made with that Savage 99. I shudda kept it.
I've since taken a bunch of whitetails with a .243, each virtually killed in their tracks. I've seen deer go down more quickly to my 100 grain bullet than deer hit with a 180 grain, 30 caliber bullet. On paper, the 30-06 is certainly more potent than the the .243. The difference is that most 180 grain bullets are intended for game larger than whitetails, are more strongly constructed and take their energy with them out the opposite side of the deer. Now, go to a 150 grain bullet in either an '06 or a .308 and well-hit deer act like they've been hit by a train.
With all due respects to your grandfather, if he insists on 30 caliber or larger for deer, he's entitled to his opinion. In the real world, as many posters have said here, it really doesn't seem to matter. Now, here's a bit of wisdom from me: deer are not hard to kill, they're hard to hit. Over the years, you'll take more deer using rifles that are easy to hit with than by using rifles easy to kill with.