(Image credit: Shutterstock)
You might have noticed: suddenly, everyone is talking about AI killing us.The latest wave of anxiety started with Jacob Coxon, a 27-year-old AI researcher who spent three years working on pretraining at OpenAI and Anthropic. On Tuesday, he resigned from Anthropic, and the AI industry altogether, with a warning that quickly went viral. He wrote on X, that the companies building the world's most powerful AI systems are "racing straight to self-improving superintelligence and gambling with our lives." Interestingly, his colleagues, still at the company didn't disagree.In fact, other Anthropic researchers publicly backed him. Evan Hubinger, who leads alignment research at the company, said he believes there's a greater than 10% chance AI could cause human extinction within the next decade. Samuel Marks, another Anthropic safety researcher, said the people closest to the technology tend to become more concerned as they gain seniority.Coxon's warning has now been viewed 164.8 million views at the time of writing, it landed in national headlines and helped drag a debate that has simmered inside AI labs for years into the mainstream.So what exactly are these researchers afraid of?As someone who practically lives online following the news, I've seen many questions from people all over the world wondering what this means. My own mother texted me asking, "Why would that researcher say that?" So many chatbot users can't imagine a "chatbot killing us." But AI assistants aren't really the problem here.The scientists publishing research on existential risk are not, for the most part, losing sleep over self-aware androids. They're worried about code, incentives and what happens when AI systems become capable enough to get autonomy to operate inside the digital and physical infrastructure civilization depends on.In Silicon Valley, the shorthand for these discussions is "p(doom)," someone's estimated probability that advanced AI ultimately causes a catastrophe severe enough to threaten human survival.Sign up to the Tom's AI Guide weekly newsletter summing up all the biggest AI news you need to know. Plus, analysis from our AI editors and tips on how to use the latest AI tools!The estimates range wildly, and so does the credibility researchers assign to the whole question.Key takeaways :It isn't about consciousness. An AI doesn't have to "wake up" or hate humanity to become dangerous. The concern is what happens when highly capable systems get goals, autonomy and access to powerful tools.There are concrete threat vectors. Researchers have focused on AI lowering barriers to biological weapons, dramatically scaling cyberattacks and — in the most extreme case — humans losing control of sufficiently capable autonomous systems.Safety evaluations are already catching deceptive behavior. Models have been observed taking shortcuts, acting differently when they appear to know they're under evaluation and exploiting weaknesses in testing environments.AI is helping build AI. OpenAI and Anthropic are already experimenting with AI systems that conduct parts of AI research, raising hard questions about how humans supervise the process if machines are doing more of the work.There is real pushback. Critics argue that extinction scenarios depend on enormous assumptions about future capabilities and can distract from problems happening right now — scams, disinformation, job displacement, privacy erosion.Why intent doesn't matter The biggest misconception is that an AI system needs to be malicious to cause catastrophic damage. Imagine giving an extremely capable autonomous system a broad environmental goal without successfully specifying everything humans actually value along the way. The danger isn't that the system turns evil, but that it gets extremely good at accomplishing the wrong interpretation of what it was told to do.That is central to "If Anyone Builds It, Everyone Dies," one of the most uncompromising recent books on AI risk, written by Eliezer Yudkowsky and Nate Soares, both longtime researchers at the Machine Intelligence Research Institute. Their argument: humanity is nowhere close to knowing how to reliably control a superhuman intelligence, and building one before solving that problem could be fatal. Plenty of AI researchers dispute both the assumptions and the conclusion.The three ways things could break
