It's the jewel in the crown of British music festivals, with a crowd of just 13,500 and a carefully curated lineup. But it almost never happened, reveals founder Simon Taffe