Guilt not going away

My sons are about 17 months apart (22 months and 6 months). I had tremendous guilt while pregnant, guilt while my second was a newborn, and all this time I was hoping it would go away at some point but it’s just not. Everyday after I put the kids to bed I feel incredibly guilty and sad for my toddler, because our time together has basically been robbed by my infant. I love my 6 month old, but he’s been an extremely difficult baby (colic, CMPA, won’t let me put him down, etc), and I feel like I’m not doing enough with my toddler. He’s the absolute sweetest boy, so kind and gentle, and he deserves so much more than what I’m giving him. When does the guilt go away? I’m tired of spending every day and night feeling guilty and sad. I miss the days where it was just me and him. But then of course I feel guilty for my 6 month old that he has a mom (me) who thinks this way about him. Like my entire waking day is just an endless stream of guilt.