What Changed
The release of Kimi K3 represents a significant shift in the landscape of open-weights large language models. With a total parameter count of 2.8 trillion, Kimi K3 positions itself as a heavyweight contender in the frontier intelligence space. The most notable shift from the previous Kimi K2 model is the reported 2.5x improvement in overall scaling efficiency. This gain is not merely the result of increased compute, but rather a combination of architectural refinements and data-recipe optimization. By transitioning to a more sophisticated Mixture-of-Experts (MoE) design, the Kimi Team has managed to maintain high performance while keeping the active parameter count at 104 billion, balancing the need for deep reasoning capabilities with the practicalities of inference latency and resource management.
Technical Details
At the core of Kimi K3 lies a complex MoE architecture supported by several proprietary innovations. The model utilizes a Stable LatentMoE configuration, which routes tokens across 896 experts, activating 16 experts per token. This sparse activation strategy is critical for managing the computational load of a 2.8T parameter model.
Two primary architectural pillars drive the performance of Kimi K3: Kimi Delta Attention (KDA) and Attention Residuals. KDA is designed to improve information flow across both sequence length and model depth, addressing the common degradation issues found in extremely deep transformer architectures. By optimizing how attention is computed and propagated, the model maintains coherence over its 1-million-token context window, a feature that is increasingly vital for long-horizon agentic tasks.
