Neck and back pain

We had a 10.5 lb baby, and now at 4 months, he is around 20 lbs. obviously all babies get to this size eventually, but for us it has hit fast and it seems there’s no truly comfortable way to hold or carry him for more than 5 minutes. It’s taken a huge toll on my neck and back, I feel like both are always sore.

I’m no gym rat, but I am in relatively good shape. So I’m kinda surprised I’m struggling so much with this. Any tips? How do I survive without long term irreversible damage? lol