Unable to use Trezor T on Qubes OS

Hi, I'm unable to use the Trezor T on Qubes OS.

First, it works fine on a Ubuntu 16.04 system (after updating the bootloader) so I assume it's not a hw issue.

I've set up the device on the Ubuntu box. When I connect the Trezor to the Qubes PC, the usb device shows up after entering the correct pin.
I start the passthrough to my (debian9) VM which seems to work fine at first. I have the correct udev rules installed and chromium finds the device after starting the passthrough. When I select it and try to connect, the beta wallet shows "Action was interrupted Error details: The device was disconnected".

dmesg shows a bunch of
"Cannot enable. Maybe the USB cable is bad?" errors, then a
"USB disconnect, device number 3", then a lot more of
"Cannot enable. Maybe the USB cable is bad?" and some
"unable to enumerate USB device".

What trickery is the Trezor doing here behind the scenes? Maybe another reset after chromium aquires the connection? Could this be avoided on the first connection (should not be neccessary on the first connection right?).