Quest VR version. Host being kicked

I've been playing co-op mode on a Quest 3. Whenever I take my headset off for longer than like 2 minutes kicks me and everyone else from the game. Is there a way to fix this?