I have been running a multi-agent development workflow for months now — one model writes the design, another generates the code, a third reviews the implementation, and I approve the result. It works well most of the time. But recently, three failures went through this pipeline undetected, and they share a pattern I had not been able to articulate until all three were on the table.
They are not impressive bugs. The individual root causes are straightforward once you see them. What makes them worth writing about is what they reveal about the structure of this kind of failure — and what kind of verification could have caught them before they reached production.
Case 1: The retry loop that never fired
An ETL pipeline synced data from a third-party ERP API to a PostgreSQL warehouse. Every 5 to 15 minutes, it pulled incremental changes using a session-based authentication mechanism. The code had a standard retry loop: three attempts, exponential backoff, and a resetLogin() call on failure to re-authenticate before the next attempt. It had worked for weeks.
Then one afternoon, every sync job started failing with the same error: COPY from stdin failed: expected N values, got 1. The pipeline reported the error and retried. And retried. And kept failing, for hours, until someone noticed.
