Nearly every AI agent benchmark you read is unfalsifiable.
Not wrong, necessarily - unfalsifiable. There's a blog post with a bar chart, a claim that framework A beat framework B, and no way for you to check it. No run count. No model version. No raw output. Often no cost. You are asked to trust a summary statistic produced by people with an interest in the result.
We publish agent benchmarks, so this is our problem too. This post is about the evidence bundle we settled on, and how you can pull one down and take it apart in about two minutes. Every command below is one I actually ran while writing this, with its real output pasted in.
The claim we're going to try to break
From one of our pilot runs: LangGraph 1.2.9 and Pydantic AI 2.13.0 both completed 20 of 20 tasks under gpt-4o, at a total spend of $0.094275.
