The former PM’s legacy won’t be getting Brexit done, or the Olympics, but allowing unsustainable numbers to come to Britain