A few months ago, I shipped my first "production-ready" AI application.
It worked beautifully — right up until real users showed up.
The model wasn't the problem. The answers were good, retrieval was solid, prompts were tight. But the app felt slow, flaky, and expensive. Some requests took 15 seconds. Others just timed out. A small traffic spike backed everything up behind a handful of long-running calls. Costs crept up for reasons I couldn't explain.
My first instinct was to blame the LLM.
I was wrong.
