After launching games that alter brightness when I alt-tab or exit.. that brightness setting stays forced

I'm one of those guys that likes his setup perfect. Everything throws me off. I'm a big gamer.. I play every day. Let's keep this short. Basically when I launch a game like WoW or CS:GO both of my monitors get brighter (looks more like increased gamma than anything). I'm fine with the games being brighter while I play them.. that's why they have gamma options... but when I alt-tab or exit these games.. that gamma is enforced onto my desktop and I absolutely hate how it looks. I feel like this shouldn't be an issue. I can't find any solutions online. I can get normal brightness settings back if I change the gamma through NVIDIA control panel and change it back or whatever... which I shouldn't have to do. It seems to happen with most games. Yes I play all these games fullscreen because I also really dislike playing games in Window mode or Window Borderless. I avoid these options.

If anyone knows any solutions.. or if this is not the proper place to post this and you know a better place please let me know. I greatly appreciate any feedback for this annoying issue.

Some specs you may want: Windows 10, GTX 970. My NVIDIA drivers are up to date. This happens when I have "Other applications control color settings" checked in NVIDIA control panel, and it also happens if I have the other option checked as well. Thanks.