Is complaining about accents racist when there’s no actual language barrier?

So I came to a realization recently. I’ve never heard a person of color complain about not being able to understand someone because of their accent. I hear white people complain about that almost exclusively. Is it racist for them to do that? I can be in the same room or conversation with them and understand clear as day and they’ll swear they can’t. I’m not even talking about accents similar to my own. I’m a native English speaker and the discussions typically take place with people with Indian or Hispanic accents. My thought is, if enough people are complaining about not being able to understand someone with an accent, it will make it harder for people with accents to get jobs, which is why I would think of it as implicit bias leading to systemic racism. Is it racist or am I overthinking it?