I feel so guilty for wanting a divorce

Warning: this is LONG so I’ll give a short summary for those who don’t want to read: me and my husband have been fighting since we got married there’s been abuse towards me and the baby. As well as the baby not being his from a short separation, and I want to leave but I’m scared I’m making a bad decision because he says he’s trying to get better.

Okay, long story: So I’ve been wanting a divorce for about the past 2 years maybe more, we’ve been married for 4. (For reference he was my first boyfriend and we were 19 and 20 at the wedding) At the beginning of our relationship he was abusive, physically and emotionally, and I want an enjoyable wife to be around either. I had horrible health issues that kept me in bed most of the time as well as bad depression which made me very sad and lethargic when I was able to be awake. We never had a honeymoon phase, he was angry with me on our honeymoon because I was sleeping a lot while I had the flu plus the other issues I talked about. So we really were off to a bad start along with a lot of other issues from when we were dating which I won’t list here because I think it’s irrelevant what happened when we were only 16-18 at the time, but the one thing I will mention is he admitted to doing a gross act to a picture of my mom. (I mention this because it’s relevant later) But anyways after multiple years of us fighting constantly and not getting along I decided I wanted to leave and to be honest I did it really horribly and separated and found someone else within a week, I feel really bad about this now but at the time I felt like I just didn’t know how else he would let me go. I ended up pregnant with that person after about a month of being separated. My husband still wanted to get back together and told me he wanted to keep the baby even though it wasn’t his and just live as a happy family whatever problems it may cause. (Like as in custody issues) the custody issue turned out to not be a problem because the biological father said he didn’t want any children and cut contact with me. I apologized everyday for a long time and still too this day we have talks about this because it’s something that really caused hurt for him and guilt on my end. Anyways fast forward to the baby being here, my husband realizes he resents the baby because it’s not his and it constantly angry and going back to old ways (throwing and slamming things, screaming, etc) and I don’t blame him at all because I think I would have been so angry if the roles were reversed and I feel so bad for what I did to him. But I was still going through post partum depression and having that around the baby was scary to me especially because he had laid hands on me in the past and I didn’t want that for my son. So when my son was 4 months old we went to a DV shelter and staid for around a month before my husband attempted to take his own life. As soon as I heard that everything that had happened just kind of slipped away and all I was worried about was making sure he was okay so I dropped my son off with my mom and went to go get my husband from the hospital. Anyways my mom didn’t want the baby around my hubs and I was still very scared as well so we decided to have the baby live with her and I stay over every weekend or more if I could. My agreement with my husband while the baby was away was that he would get anger management classes so that I could feel a little bit safer about him being around the baby. I also made it clear if he didn’t want to be the baby’s dad he didn’t have to be, it’s not his responsibility. Well 7 months go by and no anger management classes were ever attended because he said he was too stressed from going to work and wanted to just relax on his days off. I tried to give him time but seeing my mom raise my son 90% of the time was such agony. (And this is where we get back to the thing he admitted to earlier. Bc that happened and my mom also found out about it there was a weird division between me and my mom that really hurt our relationship so having the baby over there was hard, as well he was too embarrassed to ever see my family again and started saying he hated them out of nowhere) So once I felt like my husband was in a better place I brought the baby back home to life with us on the condition that if he ever was aggressive toward the baby I would be gone. So here we are at present day, he is getting along great with the baby and loves how the baby likes to copy him and thinks he’s funny and they like to cuddle in the morning time. Meanwhile my mom is so sad because she moved for my dad’s work (he travels around the country) and now can’t see the baby because my husband doesn’t want me or the baby around her because he thinks it’s her fault I left to the shelter. So now I’m feeling so guilty because despite him having gotten better with the baby and no longer abusing me I still feel like I don’t want to be with him. His anger (although not directed at me anymore) stresses me out to the point I feel relief when he was gone. We also don’t have the same values about many things in life and he has no goals and just hates everything about life. I’m trying so hard to stick through it (especially because I’ve been depressed before too and want to support him in his low places) but taking care of him and the baby all by myself for the past year now has just been so overwhelming and I just want so badly to feel peaceful in my own home. I want so badly to love him the way I used to and I don’t know how. Any advice is much appreciated, I really don’t know how to process all of this and I’m scared I’m just overreacting because I know it’s normal to get angry at things and feel like life is horrible.