Fits everybody vs cotton jersey sizing?

Do fits everybody and cotton jersey shirts fit the same? I wear a small in fits everybody but the top half of the arms are a bit too loose, so I’ve been wondering if I should get an extra small or stick with a small because it fits nicely besides for that. I’m looking at getting a couple of cotton jersey long sleeves but idk what size to get.