Our agent runs unattended. No one watches it start, no one watches it finish, and the only thing most of us ever look at is the line it writes at the end: the job passed, or the job did not. That line is one bit wide, and for a long time we treated it as if it were the whole story.
It is not. Here is the specific thing we found: a retry that eventually succeeds looks identical in the summary log to a run that never had a problem at all. Same word. Same color, if your dashboard uses color. Same position in the daily digest. The run where the first write attempt failed, waited, and went through on the second try produces exactly the output as the run where everything worked the first time. The success absorbs the failure and leaves no residue.
That absorption is the point of a retry, of course. We put retries in deliberately, because transient failures are real and because waking a human for a blip that resolves itself in four seconds is a worse outcome than the blip. The retry is doing its job. The problem is not the retry. The problem is that the retry is doing its job silently, and silence is what we use to mean nothing happened.
Consider what a system looks like as it slowly degrades. On day one, zero percent of runs need a second attempt. On day forty, perhaps twelve percent do. On day ninety, half of them do, and one afternoon the second attempt fails too, and the job goes red for the first time in three months, and everyone treats it as a sudden event. It was not sudden. It had been announcing itself for weeks in a channel we were not reading, because we had not built the channel. The pass/fail bit had been flat the entire time, perfectly stable, and perfectly uninformative about the direction things were moving.
