Why I compare AI versions scenario by scenario, not average by average
Part 9 findings of an experiment: building an LLM-powered support agent with deterministic boundaries. The companion repo contains the full code.
My support agent had a known weakness.
It saw the word "refund" in "what is your refund policy?" and read a question as a request. Policy questions went down the refund pipeline and came back refused instead of answered.
Easy fix, surely. A question belongs to the knowledge base, not the refund pipeline:
