I love my cat, but I’m wondering if bringing him home is the right decision

I recently moved my cat from my family’s home to my college apartment because I really wanted him with me. He’s extremely bonded to me, and I genuinely think he’s happiest when he’s with me.
The adjustment was rough at first—he ate less, was vocal, and seemed stressed—but he’s doing much better now. He’s eating normally, using the litter box, sleeping by me, and seems to be settling in.
The problem is I’m not doing well.
I feel like I can’t stop thinking about him. Because there aren’t other cats here, I feel like I’m constantly responsible for entertaining him and keeping him happy. I love him so much, but I’m starting to resent how much of my free time and mental energy this is taking, and I feel horrible admitting that.
I’m considering bringing him back home, where he has my family and the other cats, but I’m scared that means I’m giving up. I also want to bring him with me to vet school in a couple years, so I’m worried that if this move has been hard, another move later would be unfair to him.
Has anyone brought a cat home after realizing a move wasn’t working, and then successfully moved them again later? Or am I making things worse by bringing him home and potentially moving him again?