Earlier this year a team at UIUC audited the benchmarks the text-to-SQL field ranks itself on. The numbers in the paper (Jin et al., VLDB 2026) stopped me cold: 52.8% of BIRD Mini-Dev annotations and 62.8% of Spider 2.0-Snow annotations are wrong. Not ambiguous. Wrong, as adjudicated by human SQL experts after an agent flagged them. When the authors corrected just 100 examples and re-evaluated 16 open-source agents, relative performance swung by as much as 31% and leaderboard positions moved by up to 9 places.
Every team that picked a data agent off those leaderboards was, to a measurable degree, choosing based on noise.
I build synthetic data tooling as a side quest, and my whole library is organised around one guarantee: declared numbers hold exactly in generated data. Reading that paper, it was hard to miss that the two problems are mirror images. Benchmarks struggle to attach correct answers to existing databases. My engine attaches a database to declared answers. So I spent this week building the inversion, and this post walks through the mechanism, the verification, and the parts that did not survive their own verification, because those are the most instructive.
Why annotation-after-the-fact fails structurally
