Four frontier AI coding agents took the same build. The cheapest one won.

This month, TestSprite ran four of the world’s top AI coding agents through the same ten-phase build under identical rules. The competition, called CoderCup, is still running, and two results have already upended what many engineering teams would assume. The fastest agent rarely shipped the best software. And the cheapest one built the most accurate application, at half the cost of the priciest model in the field.For teams betting their roadmap on AI coding agents, the company argues that should change how they evaluate them. The recommendation from TestSprite CEO Yunhao Jiao is to stop ranking agents by raw model power or by how fast they finish, and to rank them instead by what survives verification.For a few years, the story of AI-native development has centered on speed: write more code, faster, with fewer people. The industry’s answer to quality followed the same logic, pay for a bigger, more expensive model. CoderCup’s early data complicates both assumptions. The hard part has moved downstream, from writing code to verifying that it works. The cost of quality may be decoupling from the model’s price, and the competition offers public data that points that way.The results also speak to how the industry measures agents today. As Jiao describes it, every lab tends to grade its own homework, not by inventing private tests, but by choosing which public score to put on the slide and which run to report. The benchmarks are often real and widely used. What outsiders rarely see is how many attempts a result took, which tasks were quietly dropped, and how the rubric was framed. And even a clean, record-setting result says little about how an agent behaves inside a given codebase, under real deadlines, against real edge cases. Engineering leaders making seven-figure tooling decisions, he argues, deserve a scoreboard no vendor owns and anyone can check.TestSprite themed CoderCup around this summer’s global soccer tournament, in part because the apps the agents built do something beyond getting graded: they predict match outcomes in real time on the public record. Four frontier coding agents received an identical assignment, build the same real, working application across ten phases of rising complexity, from initial scaffolding to features that depend on everything built before them. Each agent got the same spec, constraints, and clock. TestSprite served as the neutral referee, scoring each agent with the same test engine, which the company has open-sourced as a command-line tool under the Apache 2.0 license. Every score links to a public artifact anyone can open and inspect. No single vendor gets a home-field advantage, and no agent grades its own work.Key findings in the AI agent battleThe first finding had some unexpected surprises: speed and quality had little to do with each other. The agents that closed tasks fastest finished mid-pack, with the whole field clustered tightly between 0.793 and 0.852 on composite quality. Fast completion looked like a vanity signal; it measured how quickly an agent declared victory, not whether the software held up. That is also why TestSprite didn’t lead with the composite ranking: composite folds speed back in, and speed may be among the least informative numbers to rank on. The figure that should drive a buying decision, the company argues, is accuracy: does the app actually work, and does it keep working as the build grows?Four side-by-side scorecards, each showing an agent’s prediction bracket and its composite, correctness, wall-clock, and cost scores.The second finding rewrites a comfortable assumption: the biggest, most expensive model doesn’t necessarily win. The smallest and cheapest agent in the field took the top spot on accuracy, at 0.892, at roughly half the cost of the priciest model. It won because its verification loop did the unglamorous work: it caught every break, explained what failed and why, and drove the fix before moving on. The lesson for anyone building or buying agents, in Jiao’s telling, is that reliability comes from the machinery wrapped around the model, not the model’s raw size. How the agent planned and sequenced its steps, the feedback it pulled from testing each change against the running app, the loop it used to turn failures into fixes: that machinery is what produced code that held up.The cheapest agent won on accuracy, topping correctness at 0.892, at roughly half the cost of the priciest model.The third finding is one that launch posts tend to omit: every agent regressed. Each one broke code it had already gotten working, between 31 and 57 times during the build. To TestSprite’s knowledge, this is the first public measurement of agent regression. It also speaks to a question Jiao says he hears from nearly every engineering leader: why still keep people supervising autonomous agents? Because the agents still undo their own progress, and someone has to catch it when they do. For now, that someone is a human developer.Every agent broke code it had already finished, between 31 and 57 times across the build. Here's the full public leaderboard, regression column included.One pair of numbers captures the distance between a demo and a deployment. Across CoderCup’s ten phases, the agents got the task right on the first attempt about 80% of the time. After the verification loop caught failures and drove fixes, that climbed to 94%, with only 6% of tasks left unsolved. The 80% is what a slick launch demo shows; the 94% is what a real deployment requires. Much of what decides whether a team can trust an agent happens in the gap between those two numbers.What this means for engineering leadersOne thread connects the findings: what largely decides whether a team can ship code it trusts appears to be the verification loop wrapped around the agent, more than the model chosen or the price paid for it. Four things follow for any leader choosing an AI agent this year.Measure score recovery, the result after verification, as the primary number. The first-try score shows how good an agent’s opening guess is; the recovery score shows how good it is once it has tested its own work and fixed what broke. First-try scores sell demos. Recovery scores run businesses. Make regression a first-class metric. Track how often an agent breaks code it had already gotten working, and raise that figure onto the same dashboard as velocity, coverage, and latency. An agent that ships fast and silently regresses is not a fast agent; it’s an expensive one.Distrust any benchmark a team can’t re-run itself. If the tasks are hand-picked and the scoring stays private, the number is closer to marketing than measurement. The figure worth trusting with a seven-figure decision is the one a team can reproduce on its own code. Treat verification as infrastructure, the way teams treat CI, version control, and observability. CoderCup was refereed by the TestSprite CLI, the same open-source tool any team can point at its own agents and code, so the scoreboard isn’t TestSprite’s to control. More than 100,000 development and QA teams already rely on TestSprite, according to the company, and giving them a scoreboard they can audit means their trust rests on evidence they can verify for themselves. Do these four things, TestSprite argues, and the priciest model stops looking like the safe default. In CoderCup, the cheapest agent built the most correct app, at half the cost of the priciest one. What made the difference was the loop around it, catching what the agent broke and driving the fix before moving on. Once a team can measure what survives verification, it can stop paying for a model tier it doesn’t need.VentureBeat newsroom and editorial staff were not involved in the creation of this content.

Raccontata dadev.toventurebeat.com