Most "which AI coding model is best?" debates I see devolve into vibes. Someone pastes a cherry-picked diff, someone else counters with a different cherry-picked diff, and nobody learns anything transferable. The problem isn't the models — it's that we almost never evaluate them on our code, with our constraints, using a method we could rerun tomorrow.
This article is the harness I wish more teams built before arguing. It's a small, language-agnostic evaluation loop you can point at any model you have access to — including free tiers — and get a defensible answer to a narrow question: does this model help with the tasks I actually do?
The evaluation trap
Public benchmarks (HumanEval-style tasks, leaderboard scores) measure performance on curated problems with clean specifications. Your work is rarely that. Real tasks look like:
"Add retry logic to this half-migrated HTTP client without breaking the old call sites."
