Invite complications

I am playing on xbox and it’s not allowing me to invite pc/steam players nor letting me in their lobbies. I have tried multiple times, so have they. Cross play is on for both of us so why wont it let us?