Scroll DEV this week and you'll see the same pattern: agents everywhere, and everyone arguing about which model "codes better." One recent discussion made a sharp point — metrics from sub-agents aren't comparable to main-thread metrics — and the replies were full of people realizing their own comparisons were just vibes with extra steps.
That stung, because it's true of most of us. We paste the same prompt into two models, skim the outputs, and declare a winner. That's not an evaluation; it's a coin flip with markdown.
This article builds a tiny, boring, repeatable harness: 20 fixed coding tasks, deterministic scoring where possible, structured human rubric scoring where not, and a runner you can re-execute against any OpenAI-compatible endpoint. Total setup is one Python file and one JSONL file. No framework, no database, no dashboard.
Why a harness instead of a leaderboard
Public benchmarks measure what their authors cared about. You care about your failure modes: does the model hallucinate imports in your stack? Does it write tests that pass but assert nothing? Does it mangle SQL edge cases? A 20-task suite you own answers those questions in an afternoon, and — critically — you can re-run it every time a model updates, because model behavior drifts.
