Performance Issues after upgrading my PC

I don't know if this is good place to post this issue but maybe somebody can help identify the issue. Ever since 3.26 I upgraded my PC, but instead of getting better performance, it has gotten considerably worse.

Here is a list of things that are happening:

• Textures take a while to load, invisible enemies,wisps etc for the first minute or so in a map, after that this issue goes away even after I leave the map and re-enter.

• My character is frequently not where it's supposed to be, it's kinda hard to explain but it seems like the character on the screen is a few seconds ahead of where it's actually load in the game, this also happens to enemies sometimes.

• Movement skills don't work 8/10 times. I would use something like whirling blades and it just takes me back to where I was, at this point I basically stopped using them cause they actually slow me down.

Interestingly my FPS doesn't seems to drop much except when I'm doing performance heavy stuff but the above issues happens even on areas where not much is going on.

Here are my specs:
Samsung SSD 990 EVO Plus 1TB
Intel Core i7-12700k
Nvidia GeForce RTX 3060 Ti
Kingston Fury Beast DDR5 64GB

I'm on Windows 11 and I play PoE from steam, my SSD is at 50% capacity. Here is what I have tried so far, nothing has worked:

• I have tried multiple different settings. Don't see a different between DX11 and 12, Vulcan did seem to improve it a bit but I crash very frequently while using it(every 5 minutes or so). Graphic settings seem to have no effect, it is exactly the same on lowest and highest settings for texture, shadows and water, same thing with illumination and dynamic resolution.

• I have also completely disabled sounds from the config file, doesn't seem to change anything.

• I have checked my CPU usage while playing, it pretty much never goes to 100% so it doesn't seem to be throttled by that.

• I have run PoE both with other heavy processes like browsers with a ton of tabs and with the game being one of the only things running, same performance on both instances

• I tried installing the game on an older SSD, no change

• Updating Nvidia drivers, no change

• I have tested multiple different servers for latency, doesn't seem to change much

• I have excluded PoE from windows defender scans

I'm practically at the end of my wits with this because the game is pretty much unplayable. If anybody has a different suggestion that might work that would be greatly appreciated.