Outsider reaction to dating shorter people

I know we've all had that question (at least us tall ladies) "do you have trouble finding people your height to date?" and its annoying as balls, but while I don't, or didn't, mind dating shorter, I have recently become more uncomfortable with it because the first thing everyone says is something about him being shorter than me, and it makes me feel like even more of a giant. Am I alone in this?