Am I allowed to attend school events on my ex’s parenting time?

I’m solely responsible for my 6 year old daughter’s education. We share the custody. On my ex’s parenting time day (Thursday), it was my daughter’s celebration of graduation from kindergarten. I went there attending the ceremony. My ex didn’t show up. After the ceremony, I stayed at the school playground with many other parents and kids to watch them play. I texted my ex where we were so he would know where to pick up the child. He was angry and claimed I “ removed the child “ on his day and breached the court order. His lawyer filed notice of application today to get me pay him $5000 for breaching the court order.

What was I wrong?