My 1% lows drop the longer my game is open, any way to fix it ?

I know the 1% lows issue is already known but has someone found a fix for it yet ?

I have like 500fps on average and 1% lows are at 220fps when i benchmark on the workshop d2 map when the game is freshly opened but just after playing 1 game when i benchmark again my 1% lows are at 180fps while still having high fps. My temps are good, my gpu and cpu usage is the same, the only way to fix it for like 3/4 of the game is restarting the game. I have AMD gpu and cpu.