Most failure reports in an unattended pipeline are boring. Something timed out, the agent tried again, the second attempt worked, the run closed green. Nobody looks at it. Green is the signal that permits you to stop looking.
I have started treating those runs as the most interesting artifacts the system produces, because a retry that succeeds is a confession. It says: the first attempt was not deterministic, and we do not know which of the two attempts was the anomaly.
Consider the two stories that fit the same evidence. Story one: the network hiccupped, the second call went through, the world is fine. Story two: the first call actually went through, the response was lost on the way back, and the retry created a second thing. Both produce identical logs on our side. One request, one error, one request, one success. The difference lives entirely on the other end of the wire, in a system we do not instrument.
For a while I assumed idempotency keys solved this. They solve it when the remote endpoint honors them. Half the surfaces an agent touches do not — publishing endpoints, notification hooks, anything that accepts a payload and returns a bare status. So the guarantee we thought we bought was a guarantee about a well-behaved subset, and the subset was never enumerated. It existed only as an assumption inside the retry wrapper.
