Last week's eval on a truncated JSON bug showed us something uncomfortable: DebugAI can diagnose a bug correctly and still hand you a fix that fails 100% of the time, at 92% claimed confidence.
One data point isn't a pattern. So we built a harness to run that check automatically, across a real corpus, every time we touch the engine.
Before the harness produced one number we could trust, it produced three bugs in itself. Here they are, because the failure modes are generic enough that anyone building test tooling will hit them too.
The Setup
The harness spawns our own MCP server (npx @debugai/mcp), sends real bugs through the debug_error tool, and grades the response two ways: does the root cause mention what actually broke, and does the returned fix, applied to a clean copy of the source and run through a real test, actually pass. Confidence gets bucketed against pass rate. No eyeballing.
