You Can't Unit-Test an LLM. Here's What I Built Instead.

Every team shipping an LLM feature eventually hits the same wall: the thing you built is non-deterministic, and your whole testing culture assumes it isn't. assertEqual(output, expected) is meaningless when the output is a paragraph of generated prose that will be slightly different next time.

The usual responses are both bad. One is to shrug and ship it with no verification at all: "it looked good when I tried it." The other is to test the model itself, chasing a moving target that changes every time the prompt or the weights do.

I built an internal tool recently that leans on an LLM for exactly one step, and I wanted a real answer to "how do you know it works?" that I could defend out loud. The answer turned out to have two moves. First: make the LLM's job as small as you can get away with, so most of the system stays deterministic and ordinary unit tests still apply. Second: for the irreducibly non-deterministic part that's left, build an actual eval harness, and be honest about what each layer of it can and can't catch.

Here's how that played out.

The problem the tool solves