How do we know down isn’t antimatter?

How do we actually know that the elementary particles in the standard model are matter and their opposites are antimatter? What is there to stop someone from saying the standard model got half of them wrong? Why can’t up be matter and down be antimatter or vice versa?