people killing you when you are a baby

I'm fairly new, and I really love the game so far. yesterday I wanted my baby brother to play with me but as soon as he spawned in as a baby someone started attacking him. I went to join him, also as a baby and someone started immediately attacking me too? I understand if you want to "hunt" or whatever but I think its unfair to target babies or people who are smaller then you, I mean I mainly play as minawii 💀 I AM not defending myself.