A few weeks ago a small automation I run started producing noticeably worse output. Nothing in my code had changed. No dependency updates, no config edits, no prompt tweaks. The only variable left was the model itself — the free tier I was using had been swapped or updated underneath me, and because I had no baseline recorded anywhere, I couldn't even prove it. I just had a vibe that Tuesday's summaries were worse than Friday's.
That experience pushed me to build something I should have had from the start: a snapshot regression suite for LLM outputs. Not an evaluation harness for picking a model (I wrote about that before), but a tripwire that runs on a schedule and tells me when a model I've already chosen has drifted. This post is that workflow, with runnable code.
The problem: models are mutable dependencies
We treat pinned npm packages and locked Docker digests as table stakes, but most of us consume LLMs as a floating latest tag. Hosted models get silently upgraded, quantized, re-routed, or retired. Free tiers churn even faster — providers rotate what's available, and a model name that worked last month may now resolve to something different.
If your prompts are tuned against one behavior, a silent swap is a breaking change you will never see in a changelog. The fix is the same one we apply everywhere else: record known-good behavior and diff against it.
