I’m not sure what causes it, but I think it has to do with the amount of enemies. If there’s too many, the shout simply doesn’t knock back anymore. This happens with Salty WHC, IB and Merc.
Also, in certain areas, you are guaranteed for it not to work at all. Such as boss fights. In the War Camp finally battle, shouts will not stagger anything. Lead to a complete wipe last night when all 3 of us assumed we could just shout and keep attacking. Unfortunately, not one of our shouts staggered the trash mobs and we were all down. Leaving Queen Kerillian to try and kite 6 CWs, boss, multiple maulers and trash. She may be a god among mortals, but not even she could pull it off.