AIO for wanting to keep my kids away from their dad?

There is a lot of history, and I am trying to keep it as vague as possible to protect everyone involved.

I, (stepdad) share two kids with my wife and their bio dad.
Prior to and during their divorce, the ex was violent and disruptive, resulting in two police cases against him. My wife, being a child of divorce herself, and honestly too kind, was very lenient in the divorce proceedings. After legal separation, he moved to a different state for a year and rarely contacted his children.

He would, however call my wife and yell at her, and call drunkenly at random hours.

During this time, I came into the picture. Then, conveniently, he came back. We were nice, and worked with his schedule and let him have the kids when he was able. There were weeks during the summer that he didn’t bathe the kids, had them sleep on the ground (refused to get them a bed, toothbrushes, clothes, toys, and sometimes food to keep there)

He eventually moved in with a girlfriend two hours away from us. Once she came around, the care for our kids got better.

He almost k1lled our kid by not taking him to the doctor when they couldn’t breathe all night. We drove to where they were and took the kid to the ER. They They had dangerously low O2 levels. (They have a chronic lung condition that causes them to not be able to breathe) I shudder to think of what might have happened if we hadn’t intervened.

Recently, while our kids were with their bio dad, he got angry and hurt one of our kids. He did not take the kid to the doctor, which needed to be done. We drove two hours to get the kids to take care of them.

We have had the kids since and are pursuing protective orders. The bio dad has a CPS case open.

For numerous reasons, I don’t think the kids are safe with him. We worry he won’t call us anymore if something happens while they are with him. I worry how he will handle the kids when they are bigger and harder to manage. I worry that these are major warning signs, and something very very bad will happen.

He and his girlfriend have been harassing extended family, and us directly for weeks. We are building a legal case and so is he.

I dunno, I am willing to go for the jugular and try to get complete custody. I would love for him to never see them again, and never speak to us again. I (personally) want him totally out of the picture, I am tired of him making my wife cry. I am worried he may try to take the kids out of school without our knowledge or try to take them in another way.
I absolutely hate that he knows what cars we drive, where we live and where we work.

Is that too harsh? Am I overreacting and letting my anxiety and loathing get in the way of my perception of this? Is this guy an actual BIG issue as I perceive him, or am I just being overprotective?

I obviously will defer to whatever my wife wants, as it is ultimately her choice. But I still have feelings and opinions about it.