LLM applications fail silently at the semantic level. Standard unit tests verify that functions return values, but they cannot detect if the output is factually wrong, off-tone, or missing steps. Evals fix this by running a prompt, inspecting the output, and programmatically or judgmentally deciding its quality.
Why LLM Testing is Different
Unit tests break on non-determinism. Temperature-driven randomness means identical inputs produce varying valid outputs, which defeats static equality checks. Semantic equivalence ("Paris is the capital" vs. "Capital is Paris") makes string matching useless. Small model or prompt updates cause gradual quality drift that only surfaces at scale. Tone and accuracy require judgment that basic assertions cannot encode.
The Three Eval Types
1. Heuristic Evals check measurable properties: JSON validity, word counts, PII presence. They are fast and objective — ideal for regression gates on every pull request.
