As usual I'm late to get in on the conversation. My vote is for the 44 Mag. Its mid-range loads are plenty for deer (out to 100 yards or so) and it can be loaded up both in powder and bullet for bigger stuff. Its recoil, while stout, can be gotten used to especially if you're a reloader. Its inherently accurate. Its a common cartridge (for a handgun cartridge).
Bigger rounds like the 454 make me hurt just watching them get fired. The 357, IMHO, can not be counted on to reliably kill deer cleanly unless you're really good. The 41 is just not different enough from the 44, but also not very common, to be a better choice than the 44, IMO.