Never trust an LLM's output directly. Here's the validation layer I put on every agent.

Here's a failure mode I've seen in nearly every AI agent codebase I've reviewed: the agent receives a model response, trusts the JSON it contains, and calls .result.items[0].id — which throws Cannot read properties of undefined at 2 AM because the model returned {"result": null} on an edge case.

The model didn't hallucinate the content. It hallucinated the structure.

This is surprisingly common, and the fix isn't "use a better prompt." The fix is a validation layer that runs between the raw model output and the code that acts on it.

Why structured output isn't enough

Claude and GPT-4 both support structured output modes that constrain the model to emit valid JSON matching a given schema. This is genuinely useful and you should use it. But it doesn't fully solve the problem, for two reasons:

The model didn't hallucinate the content. It hallucinated the structure.

This is surprisingly common, and the fix isn't "use a better prompt." The fix is a validation layer that runs between the raw model output and the code that acts on it.

Why structured output isn't enough

Never trust an LLM's output directly. Here's the validation layer I put on every agent.

Never trust an LLM's output directly. Here's the validation layer I put on every agent.

Related reading

Your LLM Cannot Tell When It Is Wrong, Build for That

Your LLM didn't fail. Your application trusted it too much.

Trusting an LLM's JSON in production

The LLM Is Not the Final Authority: Building Trust Infrastructure for AI Agents

The Safety Feature That Taught an LLM to Lie

Overcoming LLM Limitations

Related reading

Your LLM Cannot Tell When It Is Wrong, Build for That

Your LLM didn't fail. Your application trusted it too much.

Trusting an LLM's JSON in production

The LLM Is Not the Final Authority: Building Trust Infrastructure for AI Agents

The Safety Feature That Taught an LLM to Lie

Overcoming LLM Limitations