Why do opticians advise you keep contact lenses away from water, but you don't have to be careful when not wearing them?

Title. Do the contact lenses do something to your eye that makes the beasties in the water affect you more? Explain like I'm 5!