I agree with you. In fact I consider the .308 as a "smaller" caliber, just a little bit shy of what I consider a "medium" caliber, the 30-06.
I don't know what size Mule deer you hunt, but I actually consider the 25-06 a little on the light side. I've shot a couple big bucks, a 5x5 that was pushing 285 lbs, and a 4x5 that was probably 250 lbs. I know that the 25-06 is very popular for deer, but I'd want just a little bit more bullet weight. That doesn't mean your choice is wrong, you probably are a better shot that I am. I just want to drop them on the spot, or close to it. For me the right choice starts at the .270 which I think is "thee Mule deer rifle" and might end at the .300 Winchester mag. I'm not a fan of that kind of recoil, but if a buck is at 400 yards, you've certainly got the power and range to drop him. I'd use the 30-06 instead of the 300 myself because I don't like recoil that much and won't use a big gun unless I absolutely have too.
One thing I know for sure, when he's a big one, and his antlers are way up over his ears, your heart is going to be doing the "wango tango." A little more horsepower can make up for a small "buck fever" induced tremble at the "jerk" of the trigger...