It's because you know damn well what people can do to you if they know about you and find cause to hate you. You also probably know that people have plenty of cause to hate you. You don't like what they like, you like things they don't want to talk about, you fit the mainstream short description of some mainstream media boogeymen, and most likely, you know something that would contradict the popular position at least one issue/thing/item/event but you also know what happens to those who disagree with said position...because they hate it when they're wrong. Perhaps there's other things wrong with you that you know are wrong. Maybe you've got some unfortunate and/or undeserved and/or easily misunderstood history. Or maybe none of this applies to you, and you just never really got the right actions ingrained into instinct. I don't want to speculate. I just want you to know that you're no worse than any other person for these differences and facets of your life.
I wish people were smart enough to see their own hatred, recognize it for what it is, and keep it in check. I wish people would think about how they'd want to be treated before they acted. But thinking takes time and if you take too long to make your moves in the game that is social interaction, you lose.