Racism - Why is racism looked at selectively?

Nigger is considered a racist word. It is seen as racist if you call a black man a nigger. Black men call each other niggers all the time.

So. Black people can use a word that white people can't. Isn't that racism?