Joining game infinite loading loop

Alright I've got a problem. First I'm on Xbox one. My brother and I are trying to play quick play. Every time we join a game, the one who isn't the party leader gets stuck in a loop of the loading screen going back to the main menu. If we don't play in a group neither of us have this problem. What is causing this and how do I fix it?