Matchmaking issues

When attempting to play with a specific friend, the game will not match make. I can play with anyone else in the game but this one dude I can't. Party size doesn't matter, cross play doesn't matter, console/pc doesn't matter. His WiFi is better than ours and even if he hosts it's still the same issues. We constantly get the ARMR0002 error but only when trying to start a match with him. When he leaves the party, the 2 of us left in the party are able to start a game with no issue.

Any ideas on what to do? We're all on Xbox.