Asked to leave a “Holiday celebration”

I work for a healthcare institution and often have “meetings” for said healthcare institution. This particular month it wasn’t a meeting, it was set to be a holiday celebration. I being a new mom, I’m not able to take my baby to daycare for 2 hours and my other childcare option has been sick, so I figured it would be okay to take my baby with me to a holiday celebration it isn’t really work related and it’s on my day off. Well apparently my direct supervisor was contacted about this and I was called in the middle of this celebration and was instructed to remove myself and my child. I was also told “we will talk about it later”. Am I wrong for thinking this would be okay? Or was this absolutely blown out of proportion?

Side note: my baby is 5 months old and was quietly sitting in my lap and wasn’t disturbing anything while we were playing Christmas related games (I.e. word scramble, quote guessing game).