They say it only takes one sperm to fertilize an egg. So what causes women to not get pregnant when sperm is ejaculated in them?

They say it only takes one sperm to fertilize an egg. So what causes women to not get pregnant when sperm is ejaculated in them? I heard million of sperm comes out, so shouldn't it be easy for women to get pregnant? But for most women it's the opposite. Why is that?