Speaking at Hay literary festival, former Labour minister says complete disengagement has potential ‘for us to end up in a very, very difficult position’