Where to draw the line between flexibility and boundaries

I try to be flexible as much as possible, and have only canceled due to lateness once. Coparent misses about 38% of visits with no communication. Is 20 minutes to hours late about 19% of the time, and changes schedule plans day of about 17% of the time. His mom does more than 20% of exchanges. By this point he sees her once a week out of his 3 visits and he is late even to those. I’d like to uphold our court order that his last minute changes or lateness that is not communicated isn’t my problem basically and I have the right to cancel, but he blows up about the time he technically is supposed to have her for and how I am inflexible and he is just going to take her anyways. Wondering if this is just how it goes or if can take this to mediation and court myself or if I need a lawyer, and will that even be enough to stop the verbal aggressiveness when he doesn’t get his way. I’m tired of my life being on pause until feels like showing up and if I try to say no it creates an aggressive and stressful argument