Can't access any of my digital games if console is not set to primary

As the title says, if my PS4 isn't set as primary I cannot access any of my digital games. Even if the PSN and PSN connections are working, if I buy a game with my main account but don't set it as primary I just cannot access it at all.

I'm not the only one with a ps4 in my family and we used to share ours "primary account" for the family sharing thing. I started to notice the problem with Man of Medan which my brother wanted to play with me, so we switched his primary account on my system and viceversa. Ever since then, we can't access our games if we don't set our PS4 as primary BUT we can access the other's games. 99% of our games are digital.. so I can pretty much play nothing at all.

Restoring licences does not help. Waiting doesn't either. Am I the only one with this problem or someone can help me maybe?