Tonic retention

When you listen to a song that modulates then changes key again, how do you determine that the new key is the original key? For example, a song starts in C, then modulates to Bb for a few bars, then changes again. How do you retain the C modality in your brain so you can recognise it as C? I’ve always had difficulty with this, my sense of tonic changes and the new key becomes the tonic.