Tencent Releases Hy3: An Open 295B Mixture-of-Experts (MoE) Model with 21B Active Parameters and 256K Context

Tencent’s Hy team released Hy3. Hy3 is a 295B-parameter Mixture-of-Experts (MoE) model. It activates only 21B parameters per token. The weights ship under the Apache License 2.0. Hy3 is aimed at reasoning, agentic workflows, and long-context tasks.

What is Hy3?

Hy3’s architecture contains a sparse MoE with 192 experts and top-8 routing. Only 8 experts fire per token, so compute stays low.

The model also uses a Multi-Token Prediction (MTP) layer. MTP predicts several tokens at once for faster decoding. Both vLLM and SGLang enable it through speculative decoding.

PropertyValueArchitectureMixture-of-Experts (MoE)Total parameters295BActivated parameters21BMTP layer parameters3.8BLayers (excluding MTP)80MTP layers1Attention heads64 (GQA, 8 KV heads, head dim 128)Hidden size4096Intermediate size13312Context length256KVocabulary size120832Experts192 experts, top-8 activatedSupported precisionsBF16