Raiding issue

Hello I just unlocked the raiding opinion and had a lot of fun with it, I destroyed all the rivals and 2 faceless ones popped up, will more then 2 come back if I wait? Don't really want to restart because I am having a lot of fun