Frontier model pre-training has converged on mixture of experts (MoE), which is fundamentally changing what limits large-scale AI training. As compute per token falls, communication increasingly determines how efficiently models scale across thousands of GPUs. NVIDIA GB300 NVL72 set a world record for pre-training DeepSeek-V3 671B at 1,648 TFLOPs per GPU, showing how advances across the entire AI platform—from silicon to networking to software—continue to push training performance forward. Every gain in pre-training efficiency means researchers can train larger models, run more experiments, and reach frontier capability faster on the same NVIDIA infrastructure.
The rapid industry shift toward these MoE architectures is driven by their massive computational efficiency. Unlike dense models, where every token activates every parameter and compute per token rises with the total parameter count, MoE models activate a subset of parameters for each token. For example, DeepSeek-V3 holds 671B parameters but activates only ~37B parameters per token, reaching frontier scale at the per-token cost of a far smaller model.
The tradeoff is communication. Those experts live on other GPUs, so every MoE layer must dispatch each token to its experts and gather the results through an all-to-all communication pattern in both the forward and backward passes. This collective sits in the critical path, making throughput as dependent on communication as on compute. Because it occurs at every layer in every training step, small delays compound until the all-to-all can no longer be hidden behind compute, and adding GPUs no longer increases throughput.
