The problem with the concept of privilege as a whole is that the vast majority of discussion about it (not necessarily represented by the discussion here) contributes to what has been called "victim culture." Put another way, it's largely just used as another excuse for someone not to get off of their ass and work instead of being put to any meaningful positive purpose.

Somewhat related, but I think that policies and academic discussion about helping people, especially in the West, is largely geared towards being a massive anti-white guilt circlejerk instead of actually getting some shit done to put people on a more reasonably even playing field.