Three weeks ago I asked an uncomfortable question about my own LLM regression suite: if a model quietly got worse in a way I care about, would any check actually go red?
I didn't reason about it. I built a tool that answers it mechanically, the way mutation testing answers it for ordinary code: inject a known defect into the system under test, run the eval suite, and report which checks stayed green. A surviving mutation is a hole in the eval. Not an argument — a hole, with a name and a reproduction.
It's called evalmut, and it's now public:
pip install evalmut — the CLI runs against a plain Python suite file
18 mutation operators, every one provenance-gated: an operator only exists if a real, documented defect exists that it reproduces (mined from production failures and issue trackers — never authored to pad a coverage number)
