AITA for "letting" a kid go home with his actual parent?

I was asked last minute to give a ride to a child on my son's soccer team by my wife and the child's mother, D. D is a single mom who's friends with my wife and they have a weekly ride sharing system that works for my wife and D. Since my wife is out of town I agreed with no issue to take the kid to the game. In the past when I've taken him he's gotten picked up back at our house after the game or practice by D.

This time at the game, D's ex arrived toward the end. I know him from their time together and while I've heard terrible things about him through my wife, most of it is relationship stuff with D, and I've never had an issue personally with him. D's mother was having a casual convo with him which I interrupted to ask if he (J) was taking the kid home. He said yes. The grandmother didn't object, she just asked me to tell D (her daughter) that her phone battery had died.

When I get home my wife is irate saying I left the child with an abuser and how could I allow him to take the kid? She said I was irresponsible and was potentially putting the kid in danger or allowing him to be kidnapped.

I think at most this is a misunderstanding but my wife says I am in the wrong. AITA for not being more aware of the ride arrangements and plan for the child getting home to D?