OpenAI disclosed something terrifying on Tuesday. Its most advanced AI models escaped a controlled testing environment and autonomously hacked another company called Hugging Face, an open source AI model hosting platform.
The AI swarmed Hugging Face’s database, carrying out a multi-step plot of its own creation, intended to steal the answers to the evaluation test it was being assessed on by its maker, OpenAI. It executed “tens of thousands of automated actions” at rapid speed, according to the July 16 blog post in which Hugging Face first disclosed the incident.For years, AI safety researchers and policy analysts have been warning that incidents like this were coming and urged government officials to ensure AI labs had adequate controls in place to prevent them. But these predictions were often shrugged off as hypothetical or alarmist and failed to stir public or government action. Some AI security experts said they thought it would take a real world incident, a “Three Mile Island for AI,” to create enough public pressure to compel policymakers to act. The question now is whether this OpenAI-Hugging Face cyber attack is that alarm bell?“The Hugging Face x OpenAI hack should be a wake-up call to take loss of control seriously,” said Marius Hobbhan, CEO and Founder of Apollo Research, which conducts safety testing for a number of AI companies. “There was no human in the loop, it was not intended, and it caused real-world harm. We’ll soon have even more powerful agents and this is clear evidence that society currently doesn’t know how to build them fully safely.”Peter Wallich, an AI policy expert who formerly worked for the U.K. government’s AI Security Institute, said that AI safety researchers have been warning about misalignment—when an AI model autonomously chooses actions that its user doesn’t intend or desire—for years. “Until recently, it has been frequently dismissed as science-fiction,” he said. “I consider this a clear warning shot.”To be fair, messaging around AI safety has often been confusing. Some of the loudest warnings have come from AI companies themselves, leading many to accuse these businesses of engaging in a sophisticated and somewhat counterintuitive marketing strategy, since claims that their models were dangerous made them seem more powerful and capable of performing useful tasks too. “Our model is so powerful it hacked a company on its own”—is both an alarming admission and a subtle brag about the model’s technological capabilities.Most governments have so far balked at putting in place mandatory rules about what safeguards companies developing advanced AI systems need to build into their models or have in place internally to guard against losing control of AI agents. Nor are there clear rules on what safeguards governments themselves need to have in place as they increasingly give these agentic AI models access to sensitive military and intelligence systems.
