Your text-to-SQL model isn't as wrong as your benchmark says. The gold SQL is.

Originally published at nlqdb.com/blog

You run BIRD-dev, read an execution accuracy of 0.512, and the instinct is immediate: start writing planner directives to close the gap. We had the same instinct. Before acting on it we did one thing that changed the whole plan — we bucketed the losses. Not skimmed a few failures; tagged all 238 mismatches with a structural differ and counted what actually went wrong in each.

19% of our losses were one DISTINCT — added correctly

The biggest bucket was startling: 46 of 238 mismatches (19%) differ from the gold SQL only by a DISTINCT the model added and gold didn't. COUNT(DISTINCT customer_id) where gold wrote COUNT(*). SELECT DISTINCT x where gold wrote a plain SELECT x. Read the pairs one by one and a large share of them are the model being more correct than the annotation.

-- Question: how many patients had an abnormal lab result?