Minister says some of those involved in anti-refugee protests are racists