Issue with enhanced input actions triggering twice

I'm working on the open/close logic of my inventory ui using common ui.

On my player bp an input action (tab key) pushes the inventory layout to a common activatable widget stack.

In the inventory widget, I have enabled "back handler" using a generic input action also mapped to the tab key.

It all works fine if I press and release the key quickly. But if I hold it down while the inventory is open, the inventory closes and then opens again due to input being handed from the menu back to the game.

Is there a way to solve this, apart from setting the widget input mode to "all" and handling the logic on my player?