My PSP is acting up.

I own a PSP E-1003 that I bought online, and I have been noticing a problem when it comes to switching the memory sticks. Everytime I go and switch one, it never reads it; and the only way to get that to work is to take it out, and blow into the socket. What could the issue be?