Will the term "gweilo" ever fade away?

We're nearly three decades into post-colonial Hong Kong and usage of the term gweilo seems to be as strong as ever. Obviously I know the origins (the Brits were pale, hence "gwei") and perhaps it was fine to use this less-than-polite term given the power differential of yesteryear.

But along the way, any non-Chinese became a gweilo and black people became haak-gwei. Some ladies called me a gweilo earlier...over the years I've heard it from colleagues, bosses.

It's been debated to death whether this term is racist. Either way, I find it offensive. Why? Because i'm half Chinese, not pale and my family roots in Hong Kong go back over a century. HK is home to generations of mixed-race people, South Asians, and families of Western descent who were born and raised in the city. Using the term based on someone’s appearance ignores their ties to Hong Kong and suggests a racial purity criteria for who qualifies as a Hong Konger.

I've also noticed when an angry "local" gets pissed off at me for some reason (e.g an accidentally bump on the sidewalk), they start hurling "sai gweilo" with particular venom.

I've heard some expats say that they don't find it offensive... frankly those people aren't in the same position to take offense. Isn't it time to retire this from the vocabulary?