AC Unity Multiplayer

A friend of mine recently got the game and we've been trying to do some free roam. Everybody has passed Sequence 3.

He can join a different friend, but I can't join them for some reason. Any idea what's going on? On Xbox Series X.