Most prompt compression discussions focus on the happy path: you have a long RAG context, you trim it to 50% of tokens, and your API bill halves. What rarely gets discussed are the failure modes: dropped system instructions, truncated JSON schemas, broken code fences, and entity names that get quietly pruned because they scored low in some similarity metric.
LLMSlim (https://www.llmslim.app) is a Python library I've spent the last few months building to handle these edge cases properly. This post goes deep on the architecture decisions that make it work.
The Core Problem: Information vs. Filler
LLM input prompts have a structural problem. The instructions that govern model behavior (system roles, JSON schemas, MUST/NEVER directives) are typically a tiny fraction of the total token count. The rest is context: retrieved documents, conversation history, background information. And within that context, a significant fraction is prose that exists for human readability, not informational density.
The challenge is distinguishing high-value informational sentences from low-value filler without a model call, without embeddings, and in under 30ms.
