I build ctxfold, a lossless prompt compressor, so I spend a lot of time thinking about the other school of prompt compression — the one that deletes things on purpose. The most important tool in that school is LLMLingua, from Microsoft Research. Before I publish a measured comparison (that post is coming), I want to explain fairly how LLMLingua actually works, because it's clever, it's genuinely effective at what it targets, and the design choice at its core is the exact opposite of mine.
The one-sentence version
LLMLingua uses a small language model to figure out which tokens in your prompt the big model could have predicted anyway — and deletes them.
That's the whole philosophical move. If a token is highly predictable from its context, removing it costs the downstream LLM very little, because the information it carried was mostly redundant. The measure of "predictable" is perplexity: tokens the small model assigns low perplexity are candidates for deletion.
The original LLMLingua (2023)
