I Ran 170 Agent Goals for $0.49. The Field Test Found 10 Issues That Unit Tests Never Would.

This is article 4 in a series about building PlannerCritic, an open-source engine where one LLM writes a plan and a second LLM reviews it. Article 1 covers the 157-goal v0.1.0 field test. Article 2 is about the critic severity bug. Article 3 is about the planner capability gap. This one is a practical guide to field testing agent systems — from 157 goals at $0.30 in v0.1.0 to 170 goals at $0.49 in v0.2.1, and what 10→0 issues across three releases teaches about release engineering.

Updated for v0.2.1: The original article covered the v0.1.0 field test — 157 goals, $0.30, 10 issues found. Since then, the engine shipped v0.2.0 (170 goals, 31 code-review bugs fixed, 0 field-test-found issues) and v0.2.1 (170 goals, 10 more code-review bugs fixed, 0 field-test-found issues, live-critic boundary evaluator, operational benchmark). This update tells the full arc: from a diagnostic tool that found 10 issues to a regression gate that finds 0 — and why 0 is the hardest result to earn.

If a field test returns 0 failures, your first instinct should be distrust. In v0.1.0, 0 failures meant our harness was silently broken. In v0.2.1, 0 field-test issues meant something very different: code review caught all 41 bugs before the LLM ran, turning the field test from a diagnostic sweep into an immutable regression gate.