Is the pull-out method really that ineffective?

My husband and I have been together for 5 years. The entire time, we have only ever used the pull-out method. We have sex probably 3-5 times per week. We are indifferent toward having children, hence only using a shoddy method. All of my friends with kids say we should have been pregnant by now. So is my husband just a pull-out pro? Can it actually be relied on to a certain extent? Or is one of us likely infertile?