For years, writes GRAHAM GRANT, the Left characterised anyone who had concerns about rising and often uncontrolled immigration as xenophobes and racists.