Originally published at nlqdb.com/blog
Our small NL-to-SQL benchmark — twenty questions, one gold query each, scored by executing the SQL and comparing result sets — came back 17/20 on a greedy pass. An immediate second run on the same commit, drawing three samples per question instead of one, came back 6/20, with 14 of the 20 questions returning no SQL at all. Ninety seconds apart, same engine, same prompts, an eleven-answer collapse.
Nothing regressed. The engine behind nlqdb runs on a chain of free-tier LLM providers, and the second run tripled the request volume the first run had already spent. The providers got tired. The score didn't measure the model's reasoning — it measured the moment the free quota ran out.
The failure signature: instant, empty, off the books
The tell is in the error tally, not the score. Those 14 no-SQL answers were circuit_open fast-fails: an earlier 429 had opened the provider's circuit breaker for its Retry-After window, so the call failed before any tokens were generated — the p50 latency of a failing question was ~0 ms. A model that reasons badly takes seconds to be wrong; a rate limit is wrong instantly. When your failures are instant, you are measuring availability, not accuracy.
