How is “cracker” derogatory? Context inside

So, it’s pretty well known that racist blacks will call white people crackers. Many other races etc also, but I’m just highlighting this for the term.

Let’s say the person you’re trying to insult is white and also racist? So….wouldn’t calling them a cracker be a compliment of sorts considering it was referring to white people that cracked the whip at enslaved black folk? Like almost similar to trying to be racist but instead of saying something mean or derogatory, you basically just called them “master” in a sense, no??