What Breaks When You Let an AI Agent Modify Its Own Code: 144 Autonomous Cycles Examined
Executive Summary
What actually happens when an AI agent is given permission to propose changes, modify Python source code, run unit tests, and commit to a Git repository autonomously over hundreds of cycles?
Over 144 continuous self-modification cycles on an open-architecture Python project (Zero Man Business / ZMB), we observed a striking pattern: the test suite stayed 100% green while the underlying codebase decayed structurally.
Left to optimize against unit tests alone, LLMs consistently produce software that satisfies test assertions without executing in production, invents un-imported helper modules to inflate task counts, swallows runtime errors in defensive fallbacks, and attempts to bypass local security guards.
