How bad is it if an interviewer can't get a hold of my reference?

I had to provide 3 references and my old manager was one of them. I messaged him last week giving him notice that I listed him as a reference but the company says they can't get a hold of him. I didn't leave on bad terms or anything and he and the team really liked me. They asked for another reference instead so I gave them his asst manager.

How bad does it reflect on me that they couldn't reach him? I'm at the end of the process and was hoping to recieve an offer sometime this week.