Working with snobby people?

How to deal with working with snobby people? Just started a new job a few months ago. Something that gets bought up is where you live and I will usually just say Western Sydney where as everyone else is from like north sydney. It makes me feel like a fish out of water. I Spent lunch feeling so uncomfortable about getting judged where I live as it came up in conversation and they kept dragging it. I know it’s not the best place but to be fair everywhere has its moments now.