I run a content pipeline where AI writes every article — and where AI is, on principle, not trusted. Before any piece ships to our site, it survives more than ninety separate verifications: research checks, fact cross-referencing, a deterministic validator with dozens of rules, integration guards. We never sat down and said "let's build an LLM evaluation harness." We sat down and said "let's not publish embarrassing text," and then kept adding checks every time something embarrassing nearly got through.
It took me surprisingly long to notice that what we'd accumulated is an evaluation harness — the same category of thing the eval frameworks and judge models are selling, just grown organically around one production system. This is a tour of what those metrics actually are, in the order they run — and of the day the whole stack taught me its own limit.
The Shape of the Stack: Code First, Judgment Last
The ninety-plus number sounds inflated until you itemize it, so let me itemize it. Before a single sentence of an article exists, the research that feeds it passes about twenty checks of its own — is the fact base structurally sound, are the sources real and reachable, is anything suspicious hiding in fetched web content. After the article is written, every factual claim in it gets cross-referenced against the researched fact sheet — around twenty more verifications, this time performed by an AI agent. Then a deterministic validator runs roughly thirty-nine atomic rules over the finished text. Finally, eight guards fire during integration into the site build. Add the self-correction re-checks and you're past ninety verifications for one article. A full subject — six articles, ten glossary entries, one hub page — crosses a thousand.
