How bad is silica cat litter?

I’ve been using silica cat litter for my 4 year old cat because it stays fresh longer and is a decent price in my local shop.

I got a bit scared after reading somewhere that it’s not good for cats so I started switching to wood, clay etc but my cat has shown repeatedly that he’s not a fan of anything but the silica - he does use the litter tray still but I can tell he’s holding it in till the last minute, he meows in distress while using the toilet and sometimes he does his business outside of the litter tray which is not like him to do.

I have had to trick him into using the litter tray by sprinkling a little bit of the leftover silica in with everything else.

To be honest it’s becoming a chore, so I’m tempted to just switch back to the silica since he’s so used to it. So I just wanted to know truly is it that bad? I see mixed views about it online - if it is definitely bad for cats then I will continue to avoid it.