Anyone else’s party chat lag when you open the game? (Xbox One)

I’m trying to play with some buddies but every time I open the game, the party chat lags so bad that we can’t understand each other. The party chat works fine when I am playing a different game, so I’m not sure if it’s just this game.