Why is my Gpu underperforming

I upgraded from an RX 580 to a 9060 XT 16GB about two weeks ago. Initially, I did notice random frame drops and weird lag spikes, but I brushed it off as shaders being compiled. Now, about two weeks later, playing the same few games, I notice that I haven't really gotten the performance upgrade I was expecting. I have a Ryzen 5 5600X as the CPU, so I don't think it's bottlenecking a lot.

For example, I was expecting about 500+ FPS on Valorant, but I see it somewhere around 300-400, but it drops to 270 as well.

I played Elden Ring on High with ray tracing medium, with weird lag spikes. I turned off the ray tracing, but still sometimes get those lag spikes.

I did, while installing the new GPU, from the AMD software, factory reset the drivers (I suppose that means it clears up any old drivers).

Anyone got a clue if this is an issue? Am I expecting more out of the card than I thought it was capable of? Are there any fixes to this (if this is fixable)?