You've hit the wall: your long-context LLM pipeline eats 89 GB of VRAM for 128k tokens, your RAG system loses the thread of a long document, and every compression approach you try trades accuracy for memory. There's a new paper that reframes the whole problem — and the fix is surprisingly elegant.

TL;DR

What: CoMem (Comprehension Memory) — arxiv:2607.28263, published July 30 2026

Key insight: Lower transformer layers handle semantic understanding; upper layers handle prediction. Cache at the boundary.

How it works: Store residual stream states at an intermediate "split layer," retrieve top-k relevant states per query, recompute only the upper layers