Sporadic frame drops when playing modded minecraft on high end PC [java]

Hi. Recently I started playing a modded minecraft server with my friends (BigChadGuy's Plus), but for some reason I am getting huge frame drops quite randomly on a pretty good pc. My friends don't have this issue and they are all playing on mid-range laptops, so I'm fairly certain it's something on my end. Over the past few days, I've tried virtually everything I can to try and solve this issue. I've allocated less and more RAM through curseforge, fiddled around with the NVIDIA control panel, lowered in-game quality settings, disabled shaders, and anything else that I have seen other people do with a sort of similar issue.

I have narrowed down the issue to happening mainly when I am staying in the same area for a few minutes (i.e. It will drop from roughly 130-150 FPS all the way down to 10-20 FPS). When I am actively exploring the world or running around, it is mostly fine and the lowest it will drop to is 80FPS. Still though, it is quite sporadic even when I'm playing normally and I feel as though this shoudn't really be happening with my PC specs.

SPECS:
Graphics: NVIDIA GeForce RTX 4070 Ti
Processor: 12th Gen Intel(R) Core(TM) i7-12700K
Ram: 32GB

Thank you, I appreciate any help.