Most comparisons of AI coding models are useless to you. Not because the authors are dishonest, but because they test on their problems: greenfield LeetCode-style prompts, demo TODO apps, or a framework you don't use. Your codebase has different failure modes — a weird build system, a legacy module nobody wants to touch, tests that take 40 minutes.
This article is a small, reproducible harness you can run in an afternoon to compare coding models against your own repository, with scoring based on your own test suite instead of vibes. The artifact is ~120 lines of shell and Python, plus a scoring rubric you can adapt.
The core idea
Instead of asking "which model is best?", ask: on a fixed set of real tasks from my repo, which model produces patches that pass my tests, fastest, with the least hand-holding?
That gives you three measurable axes:
