Freeze lag NEEDS to be fixed

My game ends up freezing for over 10 seconds sometimes, usually if someone leaves or joins. It's really annoying when I'm playing an infection based game because it'll suddenly freeze for 5, 10, sometimes even 20 seconds. It completely kills the game imo, any way to fix?