Camera app's brightness controls not always working (P9Pro XL)

Title says it all, when I use my Pixel 9 Pro XL with the stock camera app the brightness (and contrast) settings aren't always working.

It happens regardless of the photo mode or lens used, it happens with on-screen settings or with the settings in the menu, they just randomly work. Sometimes restarting the app makes it work again, then I take a picture and it stops working. But sometimes restarting the camera app doesn't change anything.

I have no clue what triggers it, it's completely random.

Anyone knows a fix for this?