How do you get the villagers to come to your island in new horizon?

I’ve talked to a few of them and invited them to my island but none of them have moved in. Is there something else in suppose to do to get them to move? Or do I have to like get to a certain milestone or something? Would love any help someone can give.