Built to catch hate. Trained to flag the word 'queer,' though not as a slur.  
The tools meant to catch harm can end up penalizing the people they were built to protect. Worth sitting with this Pride Month.  
  
For our new paper in Nature Human Behaviour, we ran 65,969 LGBTQ+ Instagram captions through Google's Perspective API and looked at which words correlated most strongly with a high "toxic" score. The top five were all identity terms: queer, gay, lgbt, sex, gender.  
  
These weren't hateful posts, because the overtly abusive content was already moderated by the platform before we collected it. What was left was the language of pride and advocacy, and the model scored it as more toxic anyway.  
This is a known failure mode: classifiers learn to associate identity words with the harassment those identities receive, then turn that association back on the community. We even trained a classifier on a dataset built specifically to detect anti-LGBTQ+ harassment. Applied to our posts, it labeled 97% of them as toxic, at 35% accuracy.  
  
Hm so? In the world of LLM-as-a-Judge and other off-the-shelf toxicity models, this is worth auditing for: identity terms can inflate toxicity scores independent of actual content. Treat the score as a biased relative signal, and please test it against your own labeled data before you let it gate anything.



[OC] "Queer" and "gay" are the words most strongly correlated with a high AI "toxicity" score in LGBTQ+ social media posts