[Fallout 4 Spoilers] More Minutemen at the Castle?

I'm around 40 hours into the game and I have 10+ settlements, but only seven minutemen at the Castle. Building a recruitment beacon brings settlers, but is there any way to get more minutemen at the castle instead of settlers?