My AI Agent Refunded a Customer Twice, and Every Status Code Was 200

Last weekend, during the Agents of SigNoz hackathon, I wanted to avoid building yet another self-healing agent or a demo that simply detected an error and retried the same workflow.

I was more interested in a harder class of failure: cases where an AI agent behaves incorrectly even though every service involved works exactly as expected.

So I built a deliberately flawed support agent that could refund ₹4,998 against a ₹2,499 duplicate charge. Nothing crashed. There was no exception, no 500, and no retry storm. Every service returned 200, and the agent confidently told the customer their problem had been solved.

On a traditional health dashboard, the entire system looked green while its behaviour was financially wrong.

This post is about what I built to catch that class of failure: Trace2Test, a control plane that compiles failed agent traces from SigNoz into deterministic, replayable regression tests.