Four ways this system goes wrong, and the code that catches each one
Part 11 findings of an experiment: building an LLM-powered support agent with deterministic boundaries. The companion repo contains the full code.
The agent asks to refund order ORD-999. There is no ORD-999. There never was.
Nothing in this codebase checks for hallucinations. The request dies anyway, in the same line of code that stops a customer reading someone else's order — a lookup that takes the authenticated session and finds nothing. An invented id and a stranger's id are the same thing to a query with a WHERE clause on the customer.
That's the pattern for every failure mode in this post. None of the mitigations ask the model to be right.
