Six weeks ago, a LangChain agent we'd deployed for a B2B client started failing on roughly 30% of its sessions. No exceptions. No 500s. Nothing in the logs that looked wrong. The agent kept running, kept returning responses, kept looking completely healthy from the outside.
We found out because the client called asking why their numbers looked off.
Two weeks of silent failures. About $2,400 in wasted LLM spend before we even knew there was a problem. And the frustrating part wasn't that we lacked visibility — we had LangSmith running. Every LLM call was traced. Every tool call was logged. We could see exactly what the agent did, step by step.
We just couldn't see that it was wrong.
The gap between "what happened" and "was it correct"
