At 04:00 UTC, every model call through our LLM gateway started returning HTTP 400. Not some calls....
At 04:00 UTC, every model call through our LLM gateway started returning
HTTP 400. Not some calls. All of them. Our tier-1 CI gate flagged it, and the
fix was committed at 04:26 UTC the same morning — about 26 minutes end to end.
This is the post-mortem.
What happened
A single misconfigured fallback line turned a $40/month API bill into $2,300 in 48 hours. Here's...
Most backend engineers already know how to handle HTTP errors. 400 means the request is bad. 401...
The first few LLM API bugs I hit in production were easy to notice. The request failed. The user saw...
Tuesday afternoon, every autonomous cycle in my agent started returning the same error: [AGENT]...
Most LLM failures are easy to describe and surprisingly hard to reproduce. A user reports that the...
April 30, 2026. 14:32 UTC. That's when DeepSeek killed one of my keys. I didn't find out until...
