Is there any way to not have high pitched whine while plugging in two audio things?

Using Bitwig on Linux, and want to plug in my soundcard (I have a couple, which are standalone recording devices such as the R16 but work as soundcards (using ALSA) and both use the computer USB as their power source.

I want to also plug in my synth (mains power) through USB so I can use midi notes in Bitwig to control the synth. The synth sound out is plugged into the in on my soundcard to record it.

Just having two things (an audio interface plus the synth (or other device), as soon as you touch the USB cable to it a high pitched whine happens.

Is there any way to not have this happen? (I don't currently have a mains powered interface, but even if I did, I think it wouldn't solve it, because it would still be plugged into my computer through USB to be the audio interface.