There is a tension at the heart of every enterprise RAG system. Better retrieval recall means more documents in the context. More documents in the context means longer prompts. Longer prompts mean higher inference cost, higher latency, and, past a certain length, degraded generation quality as the model's effective attention dilutes across too much content.
The way most teams resolve this tension is by limiting the number of retrieved documents passed to the LLM. Pull five chunks, pass five chunks. The limit is arbitrary but practically reasonable.
Context compression offers a different resolution. Instead of limiting how many documents you include, you reduce how many tokens each document contribution requires while preserving the information that is actually relevant to the specific query. The goal is to make the information denser rather than making the selection narrower.
This is not summarization. Summarization loses information indiscriminately. Context compression preserves information that is relevant to the query and discards information that is not. The output is a more efficient representation of the documents for that specific query, not a shorter version of the documents in general.
