Friends join my party, but not actual game.. Xbox One

As the title says when party members join my game there name comes up but they aren't able to join anything I join.. Anyone?