Catch LLM regressions before your users do — a tiny CI gate for LLM output

You ship an LLM feature. Weeks later you tweak a prompt, or the provider rolls the model forward under you, and something breaks — not loudly, not in a stack trace, just three answers that used to be right and now aren't. Nobody notices until a user does.

I evaluate LLM output for a living, and this is the failure mode I see most. The fix isn't a platform or a dashboard. It's treating your eval like a test: a file you run on every change, that fails the build when it should. Here's how I do it, with runnable, dependency-free code.

Make your eval a file you run

Keep your evaluation data as plain JSON so anyone on the team can edit it without touching code. Each item is a question, the known-good answer, and — the important part — how to judge it:

[

Catch LLM regressions before your users do — a tiny CI gate for LLM output

Related reading

How to Build an LLM Eval Pipeline for Your AI App in 2026

LLM Evals For Developer Tools: Useful, Correct, Safe

LLM Evaluation in Production: Building the Eval Pipeline That Runs on Every…

Writing Evals for an LLM Security Tool: How I Know It Didn't Get Worse

Mutation testing, but for LLM evals — early experiment, would love feedback

LLM Observability: Tracing, Logging, Debugging Agent Runs