Somewhere right now, a model release thread is telling you that a new checkpoint is dramatically cheaper and noticeably better than whatever you're running. Next week there will be another one. The claims aren't always wrong — that's what makes them dangerous. Sometimes the new model really is better for the benchmarks the poster ran, and quietly worse for the eleven tasks you actually do every day.
I got tired of making default-model decisions off vibes, so I built a tiny A/B harness. It takes about 30 minutes to set up, runs on a fixed set of prompts drawn from my own real work, and ends in a one-line decision: switch, stay, or split. This post is that harness, plus the decision table I use to read the results. It's a workflow, not a benchmark — your prompt set is the whole point.
Why your own prompts beat public scores
Public evals measure what eval authors care about. My day is a narrow distribution: refactoring mid-size Python diffs, writing migration scripts, summarizing error logs, and generating test scaffolding. A model can win every leaderboard and still mangle my SQL migrations, because migrations reward tedious exactness, not cleverness.
So the harness starts with 20–30 prompts I have actually sent in the last month, each with a short note on what a good answer looks like. Not formal unit tests — just enough to score consistently:
