There is a diagram I have drawn on too many whiteboards. An agent runs, produces an output, and then — right there in the request path, before the result goes anywhere — a model-as-judge scores it 8.4 out of 10 and decides whether to ship it. Everyone nods. It looks like a quality gate. It is, in fact, the single most expensive architectural mistake I see teams make with agent evals.
Here is the opinion I will defend: your real-time gate and your model-as-judge are two different systems that must live in two different places. One is a deterministic check that runs on every single execution, costs effectively nothing, returns in milliseconds, and is allowed to block the run. The other is a slow, metered, non-deterministic opinion that can only ever run offline, after the fact, on a sample. Collapsing them into one "the LLM grades the output before we return it" step gives you the worst of both: you pay judge latency and judge dollars on the hot path, and you still don't have a gate you can trust.
The fix is not a better judge. It's putting the judge where it belongs — and putting something else entirely in the path it was squatting in.
Evidence has an independence axis, not a cost axis
