Is it OK to ask white people where they came from?

I live in the USA all my life (California of all places). Is it OK for me to ask white people where they came from?

Many people ask me that all the time so I assume it's OK. But when I push back, they say they can and that's because I'm ugly and I speak with an accent. So in short, is it OK for me to ask white people where they came from? Any white people or just white people with an accent?

Thank you, and God bless.