A notorious billionaire who started building a colossal mansion and then left it to rot for four decades has finally revealed why he abandoned the project.