FPS Drops Giving Away Player Locations Through Walls

On COD, I noticed that my FPS drops when I look at a player on the other team or in the general direction of a player on the other team, even if they're through a wall. I noticed spectating a teammate in an SND on PC. 99% sure I can tell where the other players are on the map through walls by watching my FPS drop when I look in a certain direction.

I only noticed this because I was having extreme frame drops in gunfights, and it's throwing me off.