Xbox one controller misbehaving

When setting up my RetroPie I was using an xbox one controller that my brother owned and it worked great. I asked if I could borrow one of his controllers he gave me a different one but still Xbox one same color and everything and when I plugged that one into my RetroPie it won't work in Nintendo 64 or SNES but does work on GBA and Game Gear. Anyone know why my Xbox one controller might not be working on N64 now?