Recourse for late on contract?

Dear r/Homebuilding ,

Our builder is very late on a contract for a re-construction project. They started late, tore our house open for the work making it unliveable, but eventually started showing up only once a week to put in a half day of work. We're spending a lot of money on motels and airbnb's to provide a safe and sanitary environment for our baby -- cats and other animals have taken over the kitchen because the siding isn't done, our furniture is all jammed into two unrenovated rooms making them also unusable, the garage is packed with insulation waiting for them to finish so we can reinsulate, and we don't feel we can use the house. I'm just wondering whether we have any recourse.

Our contract specified an estimated start date of mid-April, an "estimated delivery date" of same, and a completion date of "3-4 weeks pending weather" with an asterisk: "completion date is dependent on weather conditions and subject to change". He started at the beginning of May and is still not finished, in spite of nearly perfect weather. Do we have grounds to withhold our lodging costs from his final payment?

Other complaints include a huge unsafe mess of old materials that has taken me 5 vacation days to clean up from on and around the house though our contract says regular site cleanup included, scaffolding left up forever although we requested it taken down, and shoddy work being repaired by another contractor. One day they made a mistake that led us to at least 25 gallons of water pouring into our roof during a rainstorm at 3am, they later promised they'd open the wall to check for damage and have two guys there every day to finish up, neither happened. Then just rolling promises of being there but something else coming up and pushing us back further.

The point isn't to bellyache but I'd be grateful for input on what we should do here, as we are very frustrated and our lives disrupted. What is worse, we need to leave town soon for a work assignment out of country and can't remain around to ensure the work is done properly -- crossing borders is difficult, risky and expensive these days. We want to be fair, the work is maybe 85% done, but we don't think we should be putting up with this any more. We have paid 2/3 in deposits at this point and would have fired them long ago, but winter is coming and in these strange times there's no way we would ever find someone else to finish the work before the cold sets in -- we live in Canada. Thanks in advance for any advice you can offer.