Every few weeks a new coding model shows up with a polished demo, and every few weeks I watch someone wire it straight into their editor because the demo looked good. Then the surprises arrive: it hallucinates an API that doesn't exist, it rewrites a file it wasn't asked to touch, or it burns an afternoon on a task a grep would have finished in thirty seconds.
The fix isn't cynicism — it's measurement. You don't need paid API access or a fancy eval suite to find out whether a model actually helps your work. You need a small, repeatable harness and an hour of honesty.
This article walks through the harness I use. Everything below is runnable, and the whole thing works with free tiers — so cost is not an excuse to skip it.
Why a harness instead of vibes
The recent discussions here about AI agents and tool boundaries circle around the same problem: we grant tools trust before we've measured what they do with it. Anecdotal evaluation has three known failure modes:
