I need help setting up the AYN Thor as a console - Specifically with an external controller

OK so in a nutshell, every so often I want to plug my Thor into my TV via HDMI-USB-C adapter.

For the most part, this is smooth and seamless. The issue I run into is that while using Retroarch to play GBA, Sega Genesis, Super Nin and so on, the controller will get mapped to a different port than before. Then if I step away or whatever, when I come back the controller has gone to sleep, I re-awaken it and now it's duplicated on 3 or 4 different ports.

I'm using an 8BITDO Pro 3 set to Switch-D

I've gone through RetroArch and set Port 1 to Thor, but every so often this switches from Thor to Odin as well but only when I'm doing this external controller nonsense (atleast so far that I can tell because I never paid that close attention before)

Anyways, I have the following:

PORT 1:

• Device Index: AYN Thor

• Device Reservation Type: Reserved

• Reserved Device for this Player: AYN Thor

PORT 2:

• Device Index: 8BitDo Pro 3

• Device Reservation Type: Reserved

• Reserved Device for this Player: 8BitDo Pro 3

How do I get it to stop adding more devices to the device index, stay static as port 2 and make this headache go away?

I hope I didn't exclude any detail, let me know if there's something I'm missing that would help diagnose the issue.

Fwiw, I trouble shot this with both GPT and Gemini, neither could resolve it