How do you do weekends with your partners?

My baby is 9 weeks old and I love him to death, he's pretty chill most of the time and only gets fussy when he wants to be fed or changed, which I'm pretty quick to do so outside of cooing, he's pretty quiet. But idk, I'd still like a day to myself sometimes 😮‍💨. My husband works and usually lets me get 2-4 hours of sleep every night before he has to get some rest. but on weekends, he's in his 'office' playing games with his friends, doing a voice-call D&D campaign for hours, or he goes out to shop/get a haircut/out to eat with his friends. I don't have a license (for medical reasons) so I can't do any of that unless he's with me (and that means baby comes along) or I'm out with a friend/family (which has only happened once out of the entire 9 weeks). Even if I was able to go out, the one time I did, he kept begging me to come home because he was so frustrated with having the baby by himself. And even when we're both home, I can't just hand him off because he gets irritated with me. (Even if I just ask him to hold the baby briefly so I can cook dinner/wash dishes, go to the bathroom, shower, etc) Anyways, rant aside, what works for you guys?