Alerting is a details game. Nobody gets paged by their intent — they get paged by the exact query they wrote. If you want to sleep through the night, the query has to mean what you think it means, and most of the time you find out it doesn't at 3am.
The clearest example I saw wasn't even subtle. A colleague had an alert on message processing count as a throughput signal for our file processor. Perfectly reasonable — if a message were a unit of work. It isn't. One message carries a batch of up to ten URLs, and the thing we actually cared about was PDF downloads.
The important part is that this was not a deliberate approximation. Nobody weighed the cost of instrumenting per-download and settled for messages as a cheaper proxy. The detail that one message holds many URLs simply wasn't known to the person writing the rule, so there was no tradeoff to weigh in the first place. We were tracking the wrong metric outright.
And everything downstream inherited it. The threshold, the SLO, the months of accumulated "is this number normal?" intuition — all of it quietly denominated in the wrong unit.
Five messages, twenty-six units of work, and no constant ratio between them.
