As party’s rise fuels fears over future visa rules, people share how the lives they have built are in jeopardy