Sean Thomas
On Tuesday, a 27-year-old British researcher called Jacob Coxon walked out of Anthropic, the San Francisco lab that has made “AI safety” its raison d’être. As he left, he announced on X that he was quitting not just the company, but the whole industry. He did this, he said, because the people building AI seriously believe it could kill us all by the end of the decade. And yet they still rush on, researching something that might extinguish our species.
In a normal industry, Coxon’s old employer might have dismissed his laments as wild scaremongering. Instead, Evan Hubinger, who actually runs Anthropic’s alignment science (i.e. the business of keeping AI safe), publicly agreed, and put the odds of mass human extinction within a decade at more than one in ten. Anthropic is trying its best, Hubinger claimed, but it has no real plan in place, as yet, for keeping any new superintelligent machine under control. Meanwhile Hubinger’s company is preparing a flotation that could value it at a trillion dollars.
It is worth reading all that again. The engineers say the machine they are making might kill everyone, even as the brokers are pricing the shares of the same engineering company. If the chief safety officer of Supernuke Inc announced that their new reactor could exterminate Homo sapiens, we would not be discussing Supernuke’s IPO.
