Originally published at tddbuddy.com.
Related reading: Agents Should Do TDD names why faithful execution of the loop matters; Your Test Suite Is Your API for Agents names the suite as interface; World-Building Is the Test Discipline Agents Need names where determinism gets designed in.
A flaky test is an annoyance to a human and a catastrophe to an agent.
The human reads a known-flaky red bar, sighs, hits re-run, and moves on. The flaky result carried no information and the human knew it. The human had out-of-band knowledge that "that integration test fails sometimes when the build server is loaded," and the knowledge let the failure get filtered out before any reaction was taken. The cost of the flaky test to the human was a few seconds of mild irritation per run.
The agent has none of that. The agent has the red bar, the test output, and the obligation to make the bar green. It does not know which failures are real and which are random because the agent does not have anyone to tell it. It cannot ask the team lead which tests are flaky this week. It cannot remember that this same test failed for the same reason last Tuesday and the fix was "wait five minutes." Every red is a signal that something is wrong. Every green is permission to stop. The contract is simple, the agent runs against it faithfully, and a flaky test corrupts the contract in a specific way that produces specific damage.
