Baby’s father in jail

Hi guys it feels kinda weird to even talk about this but it helps that it’s anonymous I guess. My babies dad went to jail when they were about a year and a half and he’s been sentenced to 5 years. We had a rough relationship when he was out and he’d always threaten me if I didn’t let them go with him. I fear that if I don’t continue to talk to him and let his family see the kids that when he gets out he will look for me and threaten me again. I don’t know if I should just take full custody and deal with the consequences when the time comes or if I should just let the kids continue to see his family and eventually him. We were 20 years old when we had them and there was some DV stuff going on as to why I never felt comfortable letting him take them. But when we split up I moved back into my dads and he’d come make a scene or threaten to make a scene and in order to give my family peace I would just go with him so that he would see them or etc. anyway when he got locked up, his family started to come to my job and would message me back to back initially I was no contact but when they started to harass my family too, I gave in and let them see them because I don’t want to disturb my family’s peace. Not sure what to do now