Last quarter I inherited a small pipeline that does something increasingly common: pull unstructured text (vendor emails, PDF invoices, changelog entries), ask a language model to extract structured fields, and insert the result into Postgres. The original author had picked a model, eyeballed twenty outputs, declared it "pretty reliable," and shipped.
Three weeks later, a silent 4 a.m. failure: the model had returned {"total": "1,240.50 USD"} — a string with a thousands separator and a currency symbol — into a numeric column. The insert failed, the retry loop failed identically, and the dead-letter queue filled up. Nobody noticed for two days because the text of the extraction looked perfect in the logs.
The fix wasn't a better model. It was treating the model's output the way we'd treat any external API response: validate it against a contract before it touches anything downstream, and continuously measure how often the contract holds. This article is the harness I built for that, why it changed which models I'll trust in pipelines, and where free model access fits in.
Why code-generation intuition fails here
Most LLM evaluation advice (including things I've written myself) targets code generation: does the function pass its tests? Extraction pipelines are a different beast in three ways:
