Lag for no reason [xbox]

My network settings are perfectly fine and I never lag in any other game. My ping was around 30-50 during the last 4 games I played but I lagged as if my ping was 500. Anyone else experience this or know why I'm lagging when I shouldnt be?