The Raiders - HOW??

So I walk into a settlement and there are typically 2-3 people, a cat and, a potato. "Oh please, rid these raiders for us they're kicking our asses!!".

So I go to the place marked on my map and there are about 40+ raiders in this giant warehouse with them all getting drunk, arguing, and having everything available from food, to gear, to weapons, to chems and alcohol. Nearby there is one settlement with two people, a cat, and a potato. For the amount of raiders in the Commonwealth they could effortlessly take over all the settlements including major ones like Diamond City and Goodneighbor.

Every raider base has more stuff and more people than entire towns. Who the hell are they raiding? How can they maintain this when there's nothing but potatoes, cats, a few messily crops around?