Refusing to pay tradesman bill question

I had someone come to my site to install some duct work. They left my site in an absolute shitfight of state and rubbish all over the place. Dust from drilling holes everywhere, empty cans, packaging pieces of off cuts, etc.

I told him he would not receive a cent until he finished the job. And that includes cleaning up. He's since made no offer to clean up, so I have not made any payments at all.

He got shirty on the phone and made vague threats. None of which were very scary. But what can he actually do? It's not a personal job - it's a work site. I'll never be on the hook for it personally, so what happens when I just refuse to give the bill to accounts and/or instruct them not to pay it?