Struggle with Xbox controller on two different setups?

So i have a two different setups (main Win 11 secondary Win10) but only one Xbox controller (Microsoft Xbox Series X Wireless Controller)

The problem is occur when i want to switch setup, then i need to "remove device" and then add it again even if it`s alredy added before to the system i trying to connect it to before it works. Windows says it is paired but the xbox button on the controller just pulsate trying to connect.

I know that you can change connection method with doubleclick on the "pair button" but it only seems to switch between bluetooth and 2.4ghz. (between consle and pc i guess) I have also updated the firmware on the controller without any luck, the Bluetooth connection got better though.

My setups is in neighbouring rooms about 6 meters (19,6 feet) apart but the problem persist even if one of the Pc´s are powered off.

I have a aftermarket dongle for the 2.4ghz that was rock solid the first couple of years but then started to drop the connection regulary.

I would really appreciate if i could switch between my setups smoothly with the same controller.

Is it a firmware restrection in the controller or just microsoft bs?

Thanks!