WIBTA if I tell my dad I wont babysit his new toddler on weekends because he keeps acting like I dont have a life

I’m 24M, work full time M-F and I’m finally making enough to move out soon. My parents divorced 3 years ago, my dad 52 just had a baby with his new wife. Kid is cute, I like him, this is not about the kid. Since he was born though my dad kind of decided I’m automatically the Saturday helper. He will text Friday night “you can come by 8 right” and if I say I have plans he gets weird and says “it’s family, you can see your friends any time.” One time he straight up dropped the baby at my mom’s house because he assumed I was there. I was not. My mom was pissed.
I helped a lot the first months because I felt bad for him, he works long hours and he is older now. But it’s been a year and he still talks like I owe him this. Last week I told him I’m going on a weekend trip with my girlfriend next month and he said “well cancel it, we’ll need you that weekend.” No please, no asking, just telling. I said no, and he hit me with “I raised you alone for years, the least you can do is help your little brother.” I get that, but I didn’t choose for him to have another kid at 52. I want to be an older brother, not a second parent with no warning. Would I be the asshole if I tell him I will help sometimes but I’m not the default babysitter anymore.