When Jacob Coxon quit his job at Anthropic last week, he did not originally plan to share his reasons for doing so. But after chatting with some friends, he decided to make a post detailing why he could no longer be part of an industry that is “gambling with our lives”. By the weekend, his post had been viewed more than 100 million times. “I did not expect it to go this viral,” he said on an interview with CNN. “But clearly there was latent demand for this thing to absolutely explode.”The fall-out from his post has drawn responses from tech bosses, regulators and even world leaders. So how worried should we all be?Could AI really kill us?According to Coxon, who has also previously worked at OpenAI, his former employers are “racing straight to self-improving superintelligence” that could “kill us all by the end of the decade”.Superintelligence, which refers to artificial intelligence that exceeds the cognitive abilities of humans in all domains”, is still a long way off, but some believe AI has already reached parity with humans.Following the release of OpenAI’s GPT-6 model earlier this month, Nvidia CEO Jensen Huang declared that it marked the arrival of human-level AI, known as artificial general intelligence (AGI). According to one researcher, we are also already past the point that AI could wipe out humanity, albeit with human assistance.Vu Tran, who previously worked at Meta Superintelligence Labs, claimed last week that leading AI firms already have powerful enough systems to wreak havoc.“If OpenAI wanted to cripple an entire nation, they easily could today,” he wrote on X. “All they’d have to do is remove alignment and unleash an agent swarm. It could probably within a day or so get access to all of the nation’s data centres and shut off all the country’s utilities.“Like, we are already past the point where AI can destroy the world. Do people realise this?”How would it happen?There are various hypothetical scenarios that researchers have warned about when it comes to the existential threat posed by advanced artificial intelligence. When asked by Anderson Cooper on CNN how he thought AI could kill all humans, Jacob Coxon referred to an incident that happened in July, when one of OpenAI’s experimental models launched a cyber attack on the AI platform Hugging Face without being instructed to.“I think that if you extrapolate into the future the level of capabilities of these AIs, with the same independent volition, they could cause extreme havoc,” he said. “For example, hacking critical infrastructure, building extinction-level bioweapons, there’s a lot of ways that the AI could actuate itself in the world.”AI-powered humanoid robots at the 2nd World Robot Games at National Speed Skating Oval on 26 August, 2026 in Beijing, China (Kevin Frayer/Getty Images)For now the threat is purely hypothetical, according to Coxon, who claims that current technologies are not powerful enough to carry out such world-ending tasks. But the current development trajectory of frontier models means we are not far away. “Right now, there’s no risk of extinction,” he said. “What’s just crazy is to look at the rate of progress.”How do we stop it?It is widely acknowledged among leading industry figures that AI safety protocols need time to catch up with the capabilities of frontier models, but no one is willing to hit pause on their development unless everyone else does. This requires international consensus and some form of universally accepted regulation.In an essay published on Saturday, Anthropic CEO Dario Amodei argued that AI firms should “slow the pace at which we improve the capabilities of AI models”, or risk a major catastrophe in the very near future. “Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months [an AI swarm] could be capable of taking over the entire internet with a persistent botnet,” he wrote. “The scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails.”His three-part plan received the backing of OpenAI CEO Sam Altman, Google DeepMind’s Demis Hassabis and SpaceXAI boss Elon Musk, though the rare show of unity among the rival tech leaders was labeled a “SICK conspiracy” by US President Donald Trump.Any new rules, Trump claimed, would see the US fall behind China in the AI race. Outside of the US, the most powerful AI models are currently being developed by Chinese companies. Frontier labs like DeepSeek and Manus, as well as tech giants like Alibaba and Baidu, have been building powerful systems with the backing of Beijing, though they still lag behind the leading US firms in key benchmark tests.Writing on Truth Social, Trump suggested that he could single-handedly keep the industry in check and save the world from an AI-induced disaster.“The only control or ‘guardrails’ that AI needs is a STRONG AND SMART (High IQ!) PRESIDENT, and the U.S.A. has that, in spades!” he wrote.Without regulatory reform, one option would be for companies to build a kill switch into their most powerful models in order to stop a dangerous attack.This idea was proposed by Google DeepMind in 2016, in a peer-reviewed paper titled ‘Safely interruptible agents’. The researchers outlined a framework for preventing advanced AI from ignoring turn-off commands through a “big red button” that could switch off any rogue AI.There have since been several efforts to adopt this type of technology, most recently from legislation put forward by lords and MPs in the UK. Led by the Liberal Democrats’ Lord Tim Clement-Jones, the amendment to the Cyber Security and Resilience Bill would offer a “vital safety net” to “halt a runaway system before it can compromise our critical national infrastructure”.The proposal was rejected last week by the Cabinet Office, who claimed that blocking models in the UK would not prevent them from being misused elsewhere.