When your career goes in the toilet, you reach for straws. This one is the straw that broke a career. I don't think he's racist, racists have better lines and comebacks. I think he just snapped, and didn't know what to say to hurt the heckler so the only thing he could think of was that. I'd bet it was the first time he said it outside of a joke.