The alert that started this
At 09:14 the dashboard for our internal AI-assisted code review service showed p95 turnaround at 41 seconds against a 60-second SLO. At 09:31 p95 was 3m 40s and the queue held 212 pending tasks. Nothing had crashed. No pod restarts, no OOMKills, no upstream 5xx spikes. The system was healthy by every readiness probe we had — and completely unusable.
The failure mode was admission, not capacity: we were accepting every coding task into a single FIFO queue regardless of queue depth, so a slow upstream model turn translated directly into head-of-line blocking for everyone behind it. Which operational action follows from that evidence? Not "scale up" — the workers were idle waiting on the model — but reject or shed work before the queue becomes the outage.
This post is the runbook we built from that morning: a capacity envelope, a local fault-injection harness, alert thresholds, and a rollback path. I ran the harness against MonkeyCode's free model access with their free server option as the backing endpoint, because it let me rehearse the whole loop without touching production quota or billing. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The harness below is endpoint-agnostic — point it at any OpenAI-compatible chat endpoint and the measurements still mean the same thing.
