Xbox Parties Not Working and Requiring Restart

I have for months and months had issues with Xbox parties just not working and requiring me and my friends to all restart our Xboxs because it randomly stops working. It's the same issue of no one can hear each other, we all leave but the party breaks making impossible to leave then when you go to start a new party it just doesn't let you at all because it says you're still in the old one. And the only fix is to fully restart the Xbox and then it works again. I am getting so sick of this for months happening on Series X and S. It's daily and a few times a day. Is there anyone who has a fix to this or a some better way to get around having to restart the Xbox? It's me and two other friends who have the issue, all three just stop being able to hear one another then the party is stuck and can't be left until all 3 consoles are shut off and turned back on. If one person doesn't they are unable to hear when joining. It's the most annoying bug or thing I've ever had and it's been months of it.