The scenario Anthropic is worried about is one where the technology stops waiting for permission. On Thursday the company argued that frontier AI developers should build a coordinated, verifiable mechanism to slow down or temporarily pause development if advanced systems begin improving themselves faster than society can manage the consequences.
The proposal is less a product announcement than a request that the industry agree on a brake before it needs one.
The trigger Anthropic names is recursive self-improvement, AI systems capable of meaningfully accelerating their own development. That capability “would be a major development in the history of technology,” the company said, but full recursive self-improvement “also might increase the risks of humans losing control over AI systems.”
As a marker of how far automation of its own work has already gone, Anthropic said that as of May, more than 80% of the code merged into its codebase was written by its model, Claude.
The argument’s sharp edge is about coordination rather than caution. A unilateral pause by one company would be easier to enact, Anthropic conceded, but would mostly just hand leadership to whoever kept going, shifting the frontier rather than slowing it.
