USB devices cause driver to crash or audio to glitch (18i8 3rd gen/Windows 11)

If I plug in or turn on a USB device, it causes the 18i8 to either have choppy audio, or no sound at all, until I change the buffer size under device settings.

Beta drivers don't seem to have helped either.