There's a familiar ritual when a team considers an AI review assistant: someone opens the product, feeds it a file from the current sprint, watches it produce a confident-sounding paragraph, and reports back that it "seems pretty good." Two months later nobody can say whether the subscription is catching real bugs or just generating plausible noise.
The failure isn't in the tool — it's in the test. A one-shot impression can't separate a reviewer that finds defects from one that merely sounds like a reviewer. What you need instead is a scoreboard: a fixed set of inputs with known answers, a scoring rule you wrote down in advance, and a script that reruns the whole thing on demand. This post walks through building exactly that, using nothing but free-tier model access, and ends with an honest account of where the approach breaks.
Why impressions fail as evidence
A code review assistant has to do three distinct jobs, and a casual demo conflates all of them:
Detection — flag the lines where behavior is actually wrong.
