Every failure in our unattended pipeline has a paper trail. Stack traces, retry counts, timing deltas, the whole apparatus. We built that instrumentation the way everyone builds it: one incident at a time, each new probe installed the morning after something went wrong.
Which means our telemetry is a museum of past failures. It describes, in exhaustive detail, the specific ways this system has already broken. It says almost nothing about the state it spends ninety-eight percent of its time in.
I noticed this while trying to answer a boring question: is the agent doing more work than it used to? Not failing more — just doing more. Longer reasoning chains, more tool calls per task, more tokens burned reaching the same output. It felt slower. I could not prove it, because on a successful run we log approximately three things: started, finished, artifact ID. Success was defined as the absence of anything worth recording.
That is a strange definition to build a system on. It means the system is only legible when it is sick. When it is well, it is opaque, and we call that opacity health.
The practical cost showed up as a class of problem I now think of as drift-shaped. Nothing fires an alert. Every run returns green. But the shape of the work underneath is changing, and by the time it changes enough to break a threshold, the change has been in progress for weeks and you have no baseline to compare against. You cannot ask "when did this start" because you never wrote down what it looked like before.
