Good fps in games only after restart.

Hi,

I just built a pc for the first time.
I noticed I was getting lower fps in games than I should when comparing with other benchmarks on youtube. Weird thing is after I restart the PC the issue fixes itself and everything seems to run as expected. If I shut it down completely however the problem returns and I need to restart again.
For example on the same spot in Control MSI says that the gpu is using 100% yet I have fps in the 50s.
After I restart I have more than double that. The core clock doesn't change in both cases.
I already tried disabling the fast boot option in windows but it didn't help. I also tried turning on nvidia power plan to always performance but it didn't help as well.
My rig is:
rtx 4070 super Gigabite Eagle OC
Ryzen 7800x3d
Corsair vengeance 6000 CL30 2x16gb
B650 Aurus elite AX v2
I am running running F20 bios and 560.81 nvidia driver.
Windows 11 home.
I am really lost here.
EDIT: Just tested after sleeping the pc - the issue returns but not as pronounced, I am getting 70 FPS in the same spot in Control.