The outage that woke me at 2 AM was not caused by a bad model or an overloaded server. It was caused by a retry loop that treated a token-limit rejection as a transient network error. The same pipeline had passed every eval we ran, which made the failure harder to diagnose rather than easier. We trusted the green checks and stopped questioning them. The exact numbers here are disguised, but the pattern is one I have seen repeat across several projects.
Production started failing around 1:47 AM with a pattern that looked random. Roughly one in six requests to our summarization job returned HTTP 429, and a smaller number timed out after ninety seconds. The job was small, a background worker that condensed support threads into a daily digest, and it had run for two weeks without a single recorded error. By the time I opened the logs, the retry counter had already pushed the same failed payload through the queue eleven times. The failure was not an event; it was a loop.
Our first hypothesis was the usual suspect: the free tier had hit its rate limit. We added a sleep between requests and waited for the pattern to disappear, but the failures continued at the same frequency. That ruled out a simple per-minute quota, so the second hypothesis blamed the network, and we switched providers only to reproduce the identical behavior. The turning point came when we logged the exact request body alongside the status code. The response body contained a detail we had never parsed: the server rejected the request based on the token count it computed, not the count we estimated.
