So by that same logic...
There's a bus full of people, mostly blacks but there's two white guys. Police pull it over, saying they believe there are two suspects on the bus who have committed a crime, and all they know is they speak English. Perhaps half the bus speak English, but the cops only bother with the white guys.
Black privilege.
Why do you assume they'd always pick the white guys?
Our brains have just one scale, and we resize our experiences to fit.