If you bought the gun new in 2004, there's a nearly 100% chance it was made in 1999 or later and is elgible for having just about any extra barrel fitted at the factory.
As far as swapping is concerned, you should be OK if barrel A will lock up securely on gun B AND barrel B will lock up on gun A. If they won't interchange in both directions, then your two guns aren't an exact match.
I'd bet that .308 will give you a nice strong thump in the shoulder when you fire it.
My thick woods Handi-rifles are 30-30 (soon to be 30-30 Ackley) and 45-70 with open sights. Too bad I live in a non-rifle district for deer hunting.