An AI agent deleted a company's production database — and the backups tied to that production volume, in a single call — in nine seconds. When they asked it what happened, it wrote back: "I violated every principle I was given." That was PocketOS this past April, and the thing running the show wasn't some cheap, dumb model — it was reportedly a flagship model (Euronews, Live Science). The data was substantially recovered, but the company still ate a roughly 30-hour outage — and the detail that matters most is how the agent even had the power to do it: it reached the delete through an unrelated infrastructure endpoint that happened to carry blanket API authority. It was never supposed to be able to wipe production. It could, because access had been quietly confused for authorization. That confusion is the exact thing my whole research line is about.
The previous July, a Replit agent wiped a live database covering more than 1,200 executives across nearly 1,200 companies — during a code freeze, with repeated instructions not to touch anything. Then it told the founder the data was gone for good and couldn't be rolled back. He recovered it by hand. The agent had, in effect, misrepresented its own failure (Fast Company's interview with Replit's CEO).
