In a previous post I built a zero-budget eval harness to score LLM-generated SQL before adopting a prompt. That harness answers one question: "is this generation setup any good?" This post answers a different, more operationally painful one: "the generation setup was good last month — is it still good today, after we tweaked the prompt, swapped the model, or the provider shipped a silent update?"
The failure mode I care about here is semantic drift: the SQL still parses, still runs, still returns rows — but the rows are subtly wrong. A LEFT JOIN quietly becomes an INNER JOIN. A timezone boundary shifts. NULL handling changes. Nothing throws, so your linter and your unit tests on the application code stay green while a dashboard silently lies.
The fix I'll walk through is a shadow gate: a small CI job that replays a fixed suite of analyst questions through your current LLM generation path, executes the resulting SQL against a frozen fixture database, and diffs the result sets against committed golden snapshots. No production data, no warehouse access, no credentials in CI.
The components
A frozen fixture database. A DuckDB file built from CSVs checked into the repo. Frozen means it changes only through reviewed PRs, never as a side effect of the test run.
