Campaigners accuse authorities of pushing ahead with new £23m pool despite ‘majority’ wanting to save historic site