Originally published on hexisteme notes.
I audited a 24-day session on a short-form video pipeline I run with a small fleet of coding agents. On paper the headline numbers were unremarkable for a project that size: 52,188 main-thread turns, roughly 1,065 turns per episode, and 1,873 calls to pytest sitting right there in the Bash tool-call log. Then the person actually using the pipeline flagged something that didn't match any of those numbers at all: the worker, they said, kept trying tests it didn't need to. That complaint didn't trace back to Bash. It traced back to a hook I'd wired in during an earlier session and half-forgotten.
Not another measurement-artifact story
It's worth being precise about what kind of failure this is, because it sits right next to a few others I've written up that look similar from a distance and aren't. A measurement proxy inserted to observe a system can quietly change what that system does, inflating the very number you added it to see. Pooling behavioral metrics across two different roles in a fleet — a long interactive session and a short one-shot worker — without separating them first can turn two nearly-identical within-role ratios into a misleading pooled headline. Two different models can bill different token counts for nearly identical input, because the token meter itself is scoped to whichever model is doing the counting. All three of those are stories about a number that comes back wrong, or a number that's right but not comparable to the number sitting next to it.
