There really is no way to answer the question as needs and abilities vary too much from person to person.
If I could have "only one" FA - and hunted with it - I would go with a 7.5" 454. Not ideal, but for "only one" it would be the most representative of the quality and power and accuracy that the guns are known for. It is a big gun, though. And the recoil is not easy to master, particularly if you are not a handloader. If you don't handload or are not committed to mastering the recoil, I wouldn't recommend a big cartridge like that.
If I was only interested in having a good all around single action - a using gun that might be hunted with but also serves other duties - no question about it, it would be a 5.5" FA 97 with adjustable sights and micarta grips in .45 Colt. Could just as easily be 44 Sp or 41 Mag, I just prefer the 45. I have express sights on mine and I do believe I prefer it over the 4-1/4" barrel length. I certainly prefer it over the 3-1/2" barrel length on my short gun, but the 3-1/2" 45 is sort of a special purpose and has its place.
Re-reading your question, if you are leaving out the 50, 475, and 454, I would definitely go with a 5.5" FA 97 .45. You could conceivably get an 83 in 44, 41, or 357, but the 97 will do all that those will do ... on a lighter frame, so the recoil will be heavier, but still.
I have a 5.5" 97-22/22WMR, a 3.5" 97-45, a 5.5" 97-45, a 4" 454, a 7.5" 454, and a 9" 454 ... a gun for every purpose. If I headed for the hills, I would take the 5.5" 97-45 on my belt and the 3.5" would be somewhere in my pack.
I highly recommend handling both models.