Buckshot kills, but it can kill very slowly. I've seen two does that needed to be shot 3 times with loads of buckshot into the chest and neck before they fell. The guys that did the shooting swore off buckshot for good, now they use slugs when hunting in shotgun only areas. The round shot has a tendency to glance off of the areas that you most want it to penetrate.
At the close ranges of an actual attack, buckshot will have no chance to spread out, so you might as well be shooting a rifle. If you think that it's impossible to miss with a shotgun when firing under stress, just go to a cowboy action shooting match. Nobody is scared, there's just a timer to provide stress, the targets are big and close, but even top shooters with customized shotguns miss a surprising amount of the time. They would have missed with a rifle, too, but the point is that a shotgun has no advantage when it comes to scoring on a critter at close range.
Round shotgun pellets don't penetrate very well compared to bullets, which may be part of the problem when it fails to stop a target. Buckshot works pretty well when you dump a load of it into an unprotected human torso at across-the-kitchen type range, especially if most of the pellets go center chest. Then again, dang near anything works when you hit center chest.
Rifles, shotguns loaded with tough, penetrating slugs, followed distantly by large bore handguns, would be my personal choices for defense against dangerous game, ranked in the order that I've listed. The 30 WCF, with 170 grain bullets, will solve any lion problem you're likely to have in the USA, if you do your part.
Heck, even my father in law was shot as close range with a shotgun full of pellets while he was a kid, and he's still around to tell the story. He's a little guy, smaller than some big toms.
As has been stated, cougar hunt from ambush, often from behind, so keeping your wits about you is your best bet. Cougar will also come to folks that are predator calling for coyotes. Once a cougar hears you make a sound that it identifies with a food source, their behavior can change radically from the usual furtive, turn-tail and run reaction to humans. They will start hunting you, and they won't necessarily stop once they've seen you. I'd want more than a .17 caliber rifle at that point, maybe that's one instance where packing a revolver along could really help out. Callers are usually set up to see the game coming, so you'd have a chance to take a big bore handgun out and take good aim with it. It's still cougar season here in Washington, so if I see one while I'm calling coyotes, I'll be sure to see how well a .30 caliber bullet stops one.