How to use different USB controller for each system?

I have a wired PS3 controller I use for PS1 and PS2 games, an NES, SNES, N64 and Sega controller I'd like to use for each system I am running. I can't for the life of me find out how to do this. I am able to use the PS3 controller to map the N64 controller when it's plugged into port 2 but when I start up an N64 game and unplug the PS3 controller and plug in the N64 controller, it's not working. I presume it's because the N64 controller is picking up the mapping for the PS3 controller which was on port 1. Is there an easier way to do this? Start up a game, plug the controller in and play or do I need to manually map the controller in the USB slot with a mouse and keyboard in once the game is running?

I have an optiplex mini pc set up to run as a little gaming console on Ubuntu, auto log in set up and retroarch set to start once the desktop loads so it's like a real console and would like to be able to simply choose a system, plug in the controller, it recognizes and that's that. Can this be done?