Oculus: Consistent loss of tracking when interacting with objects

So, whenever I pick up a magazine, weapon, or otherwise interact with a physics object, my tracking goes HAYWIRE.

No other games have this issue, SteamVR or Oculus. It's like the entire tracking system hits pause or goes random for 3-4 seconds.

Anyone else have this problem? Or a fix for it? I really want to play some meatmas!