I want to walk through three production failures on the same contract-extraction agent, because they looked unrelated at the time and turned out to be the same problem wearing different clothes. My claim, stated up front so you can disagree with it early: schema validation tells you the grammar is correct and nothing about whether the meaning is. Those are two different jobs, and most teams (mine included, for a while) only build the first one.
Case 1: valid JSON, wrong semantics
The extractor used Claude 3.5 Sonnet with Pydantic schemas. A termination_clauses field accepted list[str]. Validation passed every time. The trouble was the model returned paraphrases, not verbatim clause text, and the downstream tool did exact-string matching against a database. Paraphrases never matched.
Pydantic had no way to catch this. The schema said list[str]. Strings arrived. Valid. The fix was a second-pass semantic check (a model call with a rubric asking, in effect, "are these strings verbatim from the source?"). Success on that field moved from 61% to 94%.
Lesson: structured-output validation is syntax validation. Semantic validation is a separate layer (and you have to build it on purpose).
