If you want to understand why 2026's frontier LLMs look the way they do — why one ships Multi-head Latent Attention, another interleaves linear and full attention, a third leans on sliding windows — stop looking at benchmark scores and look at the KV cache. Almost every architectural decision at the frontier this year is, at heart, a fight with the memory that autoregressive decoding leaves behind.
This is a practitioner's map of that fight: what the KV cache actually costs, the lineage of techniques built to shrink it, and how to choose among them.
Why the KV cache dominates
During generation, a decoder-only transformer caches the key and value vectors of every past token so it doesn't recompute them each step. That cache is the price of fast decoding, and it is not small.
The size is brutally simple:
