ByteDance is reportedly gearing up to pre-train an AI model with roughly 10 trillion parameters, a scale that would dwarf every known Chinese AI system and put the company in direct competition with the most advanced Western labs. The effort would require approximately 30,000 GPUs and an estimated 3 to 6 months of continuous pre-training.
To put 10 trillion parameters in perspective, that is more than three times the size of Moonshot AI’s Kimi K3, which sits at 2.8 trillion parameters and currently ranks among the largest models produced in China. Parameters are essentially the knobs an AI model tunes during training to learn patterns in data.
What ByteDance is actually building
The model in question uses a Mixture of Experts (MoE) architecture. Rather than activating every parameter for every query, MoE models route each input to a subset of specialized “expert” sub-networks. This makes them far more efficient to run at inference time than a dense model of the same total size.
ByteDance’s Seed AI team, which reportedly consists of around 2,000 staff members, is leading the effort. The team has prior experience scaling training runs, having previously trained models up to 175 billion parameters using approximately 12,000 GPUs. ByteDance has published research on its MegaScale infrastructure, which was designed to manage distributed training systems exceeding 10,000 GPUs.
