Buckslayer,
I'm not sure about the 25-06 on deer at 500. It's not like you're going after a groundhog at that range. I think that light bullet may have run out of steam. Maybe someone that has done it and can correct me but it seems like a 120gr bullet would be kinda weak. There are many here that hesitate using a 243 at 200 and it would be way more effective than a 25-06 at 500.
I'd go 270 minimum, if it were me, and with about a 140gr or heavier bullet. The heavier bullets will drop a bit more but carry a lot more energy at that range.
Here are some data right off the Remington site. I compared the 243, 25-06 and 270. According to their data even the 270 is marginal at 500, if you use the 1000ftlbs of energy standard for deer.
Listed is the heaviest bullets available from rem for each caliber except the 270
The 243 is 100gr
The 25-06 is 115gr
and the 270 at 140gr
Velocity
Premier Core-Lokt Ultra 100 PSP CLU 2960 2709 2471 2246 2033 1832
Premier Core-Lokt Ultra 115 PSP CLU 3000 2751 2516 2293 2081 1881
Premier Core-Lokt UltraC 140 PSP CLU 2925 2667 2424 2193 1975 1771
Energy
Premier Core-Lokt Ultra 100 PSP CLU 1945 1629 1356 1120 917 745
Premier Core-Lokt Ultra 115 PSP CLU 2298 1933 1616 1342 1106 903
Premier Core-Lokt Ultra 140 PSP CLU 2659 2211 1826 1495 1212 975
Now I compared the 270, 308, and 30-06
I used the 150gr for the 270
The 165grBT with the 308
and the 165gr BT with the 30-06
Velocity
Remington Express 150 SP CL 2850 2504 2183 1886 1618 1385
Premier AccuTip 165 ABT 2700 2501 2311 2129 1956 1792
Premier AccuTipC 165 ABT 2800 2597 2403 2217 2039 1870
Energy
Remington Express 150 SP CL 2705 2087 1587 1185 872 639
Premier AccuTip 165 ABT 2670 2292 1957 1661 1401 1176
Premier AccuTip 165 ABT 2872 2470 2115 1800 1523 1281
This is using the Swift bullet in the 270, it shows the best numbers but still a little light for 500.
Velocity
Premier Safari Grade 140 SAPSP 2925 2652 2394 2152 1923 1711
Energy
Premier Safari Grade 140 SAPSP 2659 2186 1782 1439 1150 910
Sorry for the long post but this gives about the best apples to apples comparison you can get.
The 270 and 25-06 according to these numbers top out at about 400, with the 243 at about 300. Now keep in mind these are factory loads and some bullets might perform better than others but I don't think there will be a lot of variation, without a lot of velocity increase.
Hope this helped.