What’s the proper response?

Moved to Texas for an amazing job/living opportunity. Whenever I’m asked where I’m from, 9/10 I’m responded back with, “I’d hate to live there.” Today I finally had enough and replied, “we’d hate for you to live there too.” Now I’m suddenly the rude Californian. How should I have responded?