Gemma 4 ships with a context window up to 256K tokens. Every time a model lands with a headline context number, the same thing happens: people paste their entire corpus into the prompt, watch latency and memory explode, get mediocre answers, and conclude long context "doesn't work."
The window is a ceiling, not a target. Here is the mechanical reason why, and what I do instead.
Where the cost lives: attention vs. KV cache
Two different costs scale with sequence length, and people conflate them constantly.
Attention compute is the O(n²) one everybody quotes. For a sequence of length n, full self-attention compares every token to every other token. Double the input, quadruple the attention work. This dominates prefill — the pass where the model ingests your prompt.
