Should you always switch to red trinket on first back?

It seems obvious that you need to be able to clear vision for ganks and objectives, but I've also noticed that many players in my MMR do not place any stealth wards in crucial areas. This means I often don't have anything to clear, or if I do it's in a fae bush where they will see me coming regardless. On the flip side, I'm not able to contribute to vision if my team is the one lacking. Should I be keeping yellow trinket instead and just buy more control wards?