Every few weeks a new coding model launches with a price tag that makes the incumbent look embarrassing, and my feed fills up with people rewriting their configs on day one. I've been burned by this twice: once a "drop-in replacement" silently stopped emitting valid unified diffs, and once a cheaper model passed all my prompts but tripled the retry rate on long files, which erased the savings.
So now I don't evaluate new models on vibes or on leaderboard screenshots. I run a small canary suite built from my own repository history before the candidate model gets anywhere near real work. This post is that harness: the task extraction script, the runner, the decision table, and the honest limits of the approach.
This builds on a personal scorecard I've written about before, but the goal here is different — not "rank models in general," but "answer one narrow question: is this specific cheap model safe to route my traffic to?"
Step 1: Mine your own git history for tasks
Benchmarks test what benchmark authors care about. Your git history tests what you care about. I pull completed tasks straight from commit metadata:
