Every decode step loads the entire KV cache from GPU HBM. At 128K tokens that is several GB; at 1M tokens it exceeds 60GB. The bottleneck is memory bandwidth, not compute. The GPU waits while the memory bus transfers data that is likely irrelevant to the current query.
Why shared-basis sparse attention is structurally broken
Quest, ShadowKV, and RocketKV reduce this cost by estimating which KV pages matter and only reading those. They score pages by projecting the current query onto a shared low-rank basis built from the full sequence. The fundamental issue: any fixed basis W has a null space. A page whose key structure lies in that null space will score near zero regardless of its actual relevance to the query. No training procedure can fix this — it is a representational impossibility.
Proposition 1 in LOCKS formalizes it: every fixed shared projection has "page-content blind directions." ShadowKV collapsing at small token budgets is not a tuning problem; it is this structural failure showing up.
The observation LOCKS builds on
