Question about stuttering..

Why is it that whenever a person asks me the question , what? As in, "​what did you say?" I start fumbling up more and more? Like if I spoke out a clear sentence with minimal or no stuttering at all and I get asked what, it makes me panic and mess up.