Ghosted a potential client and feel bad, want to email him

I was set to work with a potential client, who then ghosted me. He wrote me an email during the holidays, apologizing for his absence and wanting to meet up. I responded to him and we were trying to arrange a time. The ball was in my court and then I got slammed with work plus going out of town for New Years and never responded to his email. I'm basically too busy to work with him at this point, but I feel terrible about not telling him this. I want to apologize to him and let him know that I got too busy and am too busy but don't want a negative relationship. Does it matter at this point, have I already burned the bridge too badly? I'm not trying to work with him but I feel like a jerk for not responding to his email and want to apologize.