Even by your own narrow reasoning for why certain people shouldn't have freedom of movement, your argument stops holding up once you look at the bottom line of how much immigrants pay into the system vs how much they take out.
I can't quickly find a statistic that filters for scary brown people countries, but as a whole immigrants contribute massively more than they take out: https://www.ft.com/content/c49043a8-...9-00144feabdc0

You're basically saying: I don't care that they pay x billions into the system if they then have the audacity to benefit from the service they pay for with their taxes!