Exclusive: Voters say they have little confidence that government can control borders despite sharp falls in net migration