Lag Spikes while looking in a specific directions (Linux)

Hello, so I've been having this issue and I'm not sure what it could be. My game runs at a stable framerate, until I look in a specific direction. Im not sure what direction that much as I don't know the map properly, I've been playing for like 3 days.

It's on my friends hosted game and it happens when I look in the general location where the underground hideout with the 3D printer and fireaxe is. It's the location with the dead guy in a bathtub. Every time i look in that general direction, every time im in a cave and i look in a certain direction it starts to lag on any graphics preset and the game drops to like 15 fps.

Inside the cave with the fireaxe (mentioned before) the lags were the worst.

Vanilla game

My specs:
Nobara linux (KDE Plasma)
i5-12600KF
32GB DDR4 3200MT/s (2x16GB)
Intel Arc B580

I can play on ultra graphics with (by eye) 60 frames or Lowest possible graphics (could be like 200-300 fps, not sure) and every time, this happens I drop to (again by eye) 15-20 fps.

Again it's only in some directions, not all the time

Anyone has an idea how to fix it?