There's a slide in every "LLM eval platform" pitch deck that says: score every response, catch every regression. It sounds responsible. It's also the fastest way to set a five-figure monthly bill on fire while measuring the wrong thing.
The mistake is treating "score every run" as a coverage requirement. It isn't. Coverage and sampling are two different axes, and conflating them is why teams end up paying a model to re-read 100% of their production traffic to produce a 7/10 that nobody reads.
Let me make the actual argument.
Evidence has an independence axis, not a cost axis
Before you can decide what to sample, you have to decide what kind of evidence you're collecting. This is where most eval thinking goes wrong: it ranks checks by cost (cheap regex vs. expensive GPT call) instead of by independence — how hard it is for the agent under test to forge the result.
