Fibroid removal before IVF?

Could it really be a small fibroid the cause of infertility ?

I have been TTC for 7 months now - had a failed IUI last month. Turns out I have 2 fibroids - one submucosal at fundus (1.8cm - 1cm) the other is smaller and not sure where. Could this be the cause of infertility? They seem too small. Doctor wants me to get them removed before proceeding with IVF - and says this is what is causing my infertility. Im 37, AMH 2.8