Here's another way of looking at it - if a black person doesn't go out of their way to hold a door for you, do you automatically assume they're a racist? If not, then applying the same sensibility to others seems fair. maybe what's unfair is to assume they think you're racist because you don't fall over yourself to be polite.