The idea if racism is such a fucking shield for minorities, it let's get away with being shitty. Hating someone for their appearance is actually retarded, and if you do it you're a brainlet. But stereotypes exist for a reason. Distrusting, or even disliking, a whole group because a large portion of them plague society, isn't an unreasonable thing. To pretend that all people are equal is to pretend that evolution stopped at the neck, and the concerned itself only with skin tone.
My point is this: if you burn crosses and obsess over nigger dick, you're probably a retard and I refuse to associate with you. But there is nothing wrong with refusing to walk alone in a predominantly black neighborhood, or not letting a syrian refugee watch your kids.