Last month I wrote about the six questions I ask every new model. Those questions are good for a first impression, but they have a blind spot: they tell me how a model talks about code, not whether it can survive contact with my code. So I built a small, ugly, reproducible harness that runs any coding model against five tasks drawn from my own git history, scores the results with my real test suite, and writes the outcome to a CSV. This post is that harness, the reasoning behind it, and where free-tier tooling fits without falling apart.
Why repo-derived tasks beat benchmark tasks
Public benchmarks (HumanEval-style) have two problems for day-to-day tool selection. First, they're saturated and contaminated — models have seen them. Second, they don't look like your work. My work is: small refactors in a Python service, tests that fail for boring reasons, and occasional SQL migrations. So instead of asking "is this model smart," I ask "does this model reduce the time I spend on the five task shapes I actually do."
The trick that makes this cheap: your git history is a labeled dataset. Every commit that fixed a bug is a task (reproduce the fix) with a built-in grader (the tests that existed at the parent commit, plus the test added in the fix commit, if any).
