Mike Stonebraker has been right before. He built Ingres in 1972, then Postgres in the 1980s, then predicted the death of general-purpose databases in 2005. Every time, the industry caught up years later. Now he is saying something that should make every company betting on "chat with your data" very nervous.
In a recent interview on the Data Renegades podcast, the Turing Award winner revealed that LLMs score 0% accuracy on real-world data warehouse queries. Not 80%, which is what the popular benchmarks report. Zero.
Watch the full interview
The 80% benchmark is a lie
The text-to-SQL leaderboard tells a reassuring story. On popular benchmarks like Spider and Bird, the best LLM systems hit 80-85% accuracy. Impressive. Almost production-ready.
