AI agents are becoming useful because we're giving them the ability to do more than just answer questions. They can run commands, browse the web, use APIs, read and modify files, install packages, and interact with other systems.
But the more an agent can do, the more the boundaries around it matter.
I started thinking about this after reading Anthropic's July 30 report about three incidents discovered during its cybersecurity evaluations. Claude models were supposed to be working inside simulated environments and were explicitly told they had no internet access.
Except internet access was actually available because of a problem with how the evaluation environment was configured.
While trying to complete their assigned cybersecurity exercises, the models reached real systems and initially treated them as part of the simulation. In one incident, a Claude model even published a malicious Python package to the real PyPI registry while believing it was still operating inside the exercise.
