Will ex's behaviour hold up divorce process?

Left my abusive ex in January 23 after being issued a protection order against him. Naturally, our (then 4yo) son stayed with me in the house.

Ex started staying in sober living in March 23 and has been there ever since.

I tried on numerous occasions to establish a regular access routine with ex and my son. Ex would never stick to it. This aggravated me as he doesn't work, living off benefits and council addiction support services. The agreement was every other Saturday, 8am to 6pm stretching to the full weekend when he was living in his own place.

He would cancel at least every other time so it became once a month due to wanting to go out with friends. He was never on time, and would text me that he was coming back early too so I'd have to drop what I was doing and head home. He cancelled one time as I needed to drive down to Manchester and couldn't guarantee I wouldn't get stuck in traffic and be a bit late back - so he would rather our son spend 5 hours in a car than see him for longer than convenient.

The last time he saw him, beginning of October 23, he showed up half an hour early, at 7:30am. I hadn't given son breakfast yet or packed a bag so sent him with a flapjack instead. New partner and I went out for the day, and due to circumstances outside our control were a little late getting back. I texted ex with 3 hours notice, said we'd be an hour late and apologised and ex began threatening to call the police, social services etc claiming I was an unfit parent, citing my son not having had breakfast or a bag packed as extra justification.

After this incident I blocked his number due to the abuse but gave him new partner's, my dad's, Facebook and email also still open to him) and no one has heard anything from him since then at all. No attempts at contact have been made.

I can apply for the conditional order for divorce on the 9th Feb but I'm concerned that the lack of contact from him is going to drag things out. I don't see why I should have to pay for a child access court fee when I've paid for the divorce, financial order and receive a meagre £30 a month in child maintenance, and when I am perfectly happy with the way things are and not having to deal with his shitty behaviour towards me.

What should I do?