In September 2024, Anthropic published Contextual Retrieval. The trick: generate a one-sentence context per chunk with an LLM and prepend it to the chunk before embedding. On their hybrid vector + BM25 setup, the top-20 retrieval failure rate drops from 5.7% to 2.9% (a 49% reduction). Add a reranker and it falls to 1.9% (67% reduction). Their published cost is around $1.02 per million document tokens, with prompt caching applied.
If your source documents have a clean heading hierarchy, the document itself gives you a usable prefix for free. No LLM call per chunk. This post is how that path looks in production, in the itrstats tax assistant, where the knowledge base lives in markdown and the retriever is hybrid pgvector + Postgres tsvector.
.md file ┌─────────┐ ┌──────────┐ ┌─────────┐ ┌────────────┐
or scraped URL ──────▶│ cleaner │───▶│ splitter │───▶│ chunker │───▶│ embedder │
└─────────┘ └──────────┘ └────┬────┘ └─────┬──────┘
