Is it common for men to feel insecure about their sexual experience?

I see guys worry about being virgins in their 20s or not having much experience. Is that something most men stress about, or do you not think it matters as much as people make it seem?