Goblins amd accidental racism

Okay, so I'm running a playtest campaign for my TTRPG, TOGWITO, and one of the plot points is that there's a goblin horde on the island that the heroes are trying to help settle.

I've had a few players make comments about "Oh this is the Obviously Evil Species and since they're Fundamentally Bad we should all be Fantasy Racist towards them," which is a rather rough thing to hear.

In-lore, Goblins are moderately intelligent, capable of communication with each other, but have shown no interest in any other species. Their society works entirely on a hierarchy of strength, with the leaders being the ones to beat the rest of the tribe in a fight. They do have Goblin Sages, who are incredibly weak but are actually wise amd attempt communication with other species (even when their tribes never listen.)

What can I do to make them seem less of "narratively justified racism targets" and more "big scary guys who are unwilling to work with anyone"?