Mutation testing, but for LLM evals — early experiment, would love feedback

I write evals for LLM features, wire them into CI, and move on. But a passing eval suite has a blind spot: it tells you nothing about whether it would actually fail if the model quietly got worse. Green isn't the same as good.

So I built muteval to measure that directly. It borrows mutation testing from software engineering — you deliberately break the thing under test and check whether your tests catch it. muteval degrades the system (weakens a prompt rule, drops a retrieved doc, swaps in a weaker model), reruns your existing eval suite against each degraded version, and reports what fraction of those injected regressions your evals caught. The ones they miss are concrete coverage gaps.

Mutation score: 33% (2/6 caught)

SURVIVED: deleted "if the answer isn't in the context, say you don't know"

Enter fullscreen mode

Mutation testing, but for LLM evals — early experiment, would love feedback

Related reading

Writing Evals for an LLM Security Tool: How I Know It Didn't Get Worse

How to Build an LLM Eval Pipeline for Your AI App in 2026

I Built an Adversarial Eval Framework and Attacked 5 LLMs — Every Single One…

Better Experiments with LLM Evals — A funnel, not a fork | Spotify Engineering

Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics

I built an LLM eval framework from scratch. Here is what I wish I had bought…