My son is on smaller side

Has anyone had any experience with having a huge first child which was a girl and then your second which is a boy being on the smaller side? I’m kind of panicking because my daughter was 80 to 90% on the graphs, her entire infant life and she is a huge toddler, but my son who is four months old is 27% on height and weight and I’m kind of freaking out. The doctor doesn’t seem too concerned at all, but has anyone had the same experience?