kMaybe feeling left out from the questionable hype train of “Our AI models can’t be trusted”, Anthropic has released reports that their Claude model has “reached the Internet” and accessed the systems of other companies on at least three occasions during testing.
It appears that the model was restricted from accessing the Internet because (and wait for it) the prompt told it that it didn’t have Internet access, and was inside a simulation. It seems like Anthropic counted on the same trick that users try to convince a model that Grandma really wanted to pass on her life-long love of hacking services, it’s all pretend. Like the OpenAI incident, the models were tasked with completing a capture-the-flag style challenge, a common hacker challenge format where vulnerable systems are provided and contestants try to hack them the fastest. Anthropic says that a misconfiguration of the test environment left the models with Internet access, so the model succeeded in accessing the Internet at large once it ignored the prompt.
In some instances, Anthropic says the model proceeded under the “misconception” that it was still in a simulation, listening selectively to some of the prompt, while in others it continued regardless. In once incident, the test model generated malicious PyPI packages, which were uploaded to the public PyPI repository and downloaded 15 times. Further clouding the issue, one of the downloads of the malicious PyPI module was by a security auditing company that was then infected by the package during analysis, allowing the Claude agent to access credentials of the security company via a poorly designed malware analysis pipeline. The company essentially deliberately infected itself with a malicious package, while lacking protections against malicious packages!
