Your RAG Answers From 62% of Your Corpus and Won't Tell You

Every retrieval system I have ever deployed has one property in common with every retrieval system you have ever deployed: it will answer. It will always answer. Ask it something that lives in a document it never ingested, and it will not say "I do not have that." It will find the nearest thing it does have, rank it, cite it, and hand it to you with the same confidence it would give a perfect match.

This is not a hallucination problem. The model behaved correctly. It answered from what it was given. The failure happened one layer down, in the boring part nobody demos: ingestion.

Last week I measured the corpus coverage of a self-hosted knowledge base I have been running for months. Nineteen hundred documents, semantic search, embeddings served locally, the whole pipeline built and monitored by me. I expected a number close to 100%.

I got 62%.

The number before the number was worse, and also wrong

Your RAG Answers From 62% of Your Corpus and Won't Tell You

Other newsrooms on this story

Related reading

Your RAG System Is Lying To You About That Table

Your RAG Retrieved the Right Documents but Still Gave the Wrong Answer

Your RAG Retrieved the Right Document — So Why Was the Answer Wrong?

Most RAG Problems Are Retrieval Problems. Here Are 8 Fixes That Worked for Me

Retrieval-Augmented Self-Recall — Part 3: Teaching RAG to Say \"I Don't Know\

RAG Evaluation: How to Know Your Retrieval Isn't Broken