Kimi K3 made headlines with 2.8 trillion parameters, but the most interesting part for learners is not the size. It is the attention mechanism.
K3 uses something called KDA-Kimi Delta Attention. It is a variant of linear attention, which is a different way of computing the relationship between tokens in a sequence. If you are learning how transformers work, understanding the difference between standard softmax attention and linear attention will help you see why architecture choices matter more than parameter counts.
Standard attention in one paragraph
In a standard transformer, every token looks at every other token. If your sequence has N tokens, the attention matrix is N x N. For a 1-million-token context window (which K3 supports), that matrix would be enormous-impossibly large to compute directly.
That is why most models use tricks like sliding windows or sparse patterns to avoid computing the full matrix.
