Originally published at nlqdb.com/blog
Every text-to-SQL engine publishes the same two numbers: BIRD and Spider. Ours are not flattering — the strict-$0 free-model chain behind nlqdb currently scores 0.52 on BIRD Mini-Dev and 0.19 on the Spider 2.0-lite SQLite subset. We track both weekly, against a pinned baseline, with a paired significance test, because those benchmarks are the honesty instrument of this field: hard, public, and comparable to every research paper.
But look at what they actually measure. BIRD's databases are real-world dumps — dozens of tables, cryptic column names, dirty values, questions that hinge on external knowledge notes. Spider 2.0 is enterprise-analytics scale on purpose; its authors built it because models had gotten too good at the small clean stuff. Both are the right kind of hard for a research leaderboard. Neither looks anything like the database a user of a product like ours ever touches.
The schema your users build is the one you never scored
Our users describe a goal in plain English and get a small, freshly-provisioned Postgres: a form-submissions table, a four-table agent-memory schema, a webhook event log. Five tables, honest column names, no fifteen-year accretion of legacy views. That shape — the one 100% of production queries actually run against — had zero rows in either benchmark. Which means the headline accuracy number described a workload we don't serve, in both directions: it undercounts what users experience, and it can hide regressions on the queries they really ask.
