RDP Sessions extremely slow network performance, tried all the fixes I know.

I have an issue that has been driving me nuts, so I am reaching out to see if someone has experienced this issue.

Single Remote Desktop Services session host for 6 users. The network throughput is brutally slow. Here's the funny part though- I can log into the RDP session and perform a bandwidth test, horrible speeds. I log into the console side by side and perform the same test and I get results I'd expect- full throttle throughput. It seems when users all log out then log in again it improves a bit, but begins crawling shortly after. It's a nuisance. I have ruled out it being a HyperV issue (the RDP server is a VM), I have disabled the offloading on the NICs (physical and virtual switches and in VM) I have all updated drivers and tried Windows drivers, a complete clean rebuild shows the same symptoms, and this is not specific to just the RDP server, but this shows on the other two servers as well using RDP to manage them.

Servers are functionally fine, Intel I350 nics, tons of RAM, plenty of processing power. No hardware issues, and the OS (server 2012 Host and VM) are all fully patched.

I haven't much hair left. Anyone have any ideas I may be missing?

EDIT- I have eliminated the networking as an issue as well as viruses, junk apps, bandwidth. It's just spooky

EDIT2- This is interesting. I have checked on a few other RDP servers I have running in different orgs and this behaviour is similar across them too. I can expect a bit of loss, but not 50% or more. All are in Hyper V. This must be something with the virtual switches?