Connected controller drops frames on PC?

My game has low FPS even at any resolution with a 6750xt graphics card, but I noticed this only happens when I use a controller. When I move the mouse and play with it, the game locks at 60 FPS, and when I use a controller, the frame rate drops drastically. Does anyone know what bug this is?