Buzzing in headphones on new PC, think it's interference. Should a USB sound card fix this?

I bought a new PC a week ago and there's been a constant buzzing in the headphones. I've tried multiple headsets, different cables/adapters etc, different ports on the PC and the audio always has a constant buzzing. The front panel audio jacks are much worse but the mobo audio jack still has an on-and-off buzzing. I figure it is interference from somewhere. I've tried unplugging everything from my PC except the headphones and it's still there - when I unplug my displayport monitor it makes a big difference to the front panel audio jacks.

Should buying an external USB sound card like this one https://www.pccasegear.com/products/46690/creative-sound-blaster-x-g1-portable-sound-card fix the issue? Any one I can get shipped locally from Australia is fine.

I haven't tried a USB headset, only my senheiser hd599's and audio technia mtx50 which both use the audio jack. Some relevant specs if anyone's curious are Gigabyte B450M wifi mATX mobo, RTX 2060 gpu, Karuza V3 case (here)

Thanks for any advice!