A free token quota is a budget, and a budget without a burn-rate measurement is just a number in a marketing email. The only honest way to evaluate a free AI coding tier is to run it against a fixed task set with controlled variables and repeatable metrics. This article defines that method and provides a harness that produces numbers worth trusting.
Recent DEV discussions about perfect harness scores and poor model scores point at the real problem, which is that evaluation quality, not model quality, is often the variable being measured. MonkeyCode, an open-source coding project, currently offers a free ten-million-token allocation and a free server option, and that offer is a useful test case for this method. A quota without a burn-rate measurement cannot be evaluated, and the free allocation makes the first benchmark run free as well.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The dataset: fixed tasks from real history
A benchmark is only as honest as its task set, and the best task set is one the model has never seen. Pull twenty small tasks from a real repository's commit history, where each task contains a prompt, a setup command, a test command, and an output filename. The prompt describes the change, the setup installs fixtures, and the test asserts the behavior, so every output either passes or fails without human judgment.
