I run a lot of RAG pipelines. And for a while, I was doing what most people do: retrieve the top-k documents, concatenate them into the context, and ship the whole thing to the LLM.
It worked. But when I started looking closely at what was actually in those contexts, I noticed something uncomfortable: a huge fraction of every prompt was filler. Background sentences that restated the obvious. Transition paragraphs that connected ideas the model didn't need connected. Prose padding that existed because technical documents are written for humans who need continuity, not for models that need information density.
And I was paying for every token of it.
The Problem Is Worse Than You Think
Here's what makes this particularly painful: it's not just a billing issue. Long-context prompts have real performance implications.
