Last month a model I'd been watching dropped its token price by half, and three people sent me the announcement within an hour. The implied question was always the same: when are you switching?
My answer, these days, is: after it survives quarantine. Because the last time I swapped a model based on announcement-day excitement, everything looked fine for nine days. Then a scheduled job started emitting subtly malformed JSON — valid enough to parse, wrong enough to corrupt downstream state — and I spent a weekend reconstructing which records had been poisoned. The money I saved on tokens wouldn't cover one hour of that cleanup.
The economics of model swaps are lopsided. The upside is small and predictable (cheaper tokens). The downside is unbounded and sneaky (behavioral regressions in edge cases your happy-path tests never exercised). So I built a pipeline that treats every new cheap model like an untrusted dependency with an attractive changelog: it gets isolated, probed, and graduated in stages. Here's the whole thing.
What the pipeline needs (and what it doesn't)
Three ingredients: candidate model access, somewhere disposable to run the evaluation, and checks that don't require a second LLM to grade the first one.
