Alibaba's Qwen team is introducing Qwen3.8-Flash-Next, a multimodal mixture-of-experts model that serves as an architecture preview of Qwen4. It aims to match much larger models at a fraction of the training cost.
The model has 125 billion total parameters but only activates 6 billion per token. It also includes 51 billion parameters in a novel N-gram embedding layer, one of the architecture innovations slated for Qwen4. This layer stores common word groups as standalone entries in a kind of "phrase dictionary" and can sit in regular system RAM rather than on the GPU at "relatively low additional cost."
The N-gram embedding layer (bottom, purple) feeds phrase-level information into the start of the network. It accounts for 51 billion parameters but runs in system RAM instead of GPU memory. | Image: Alibaba / Qwen
The model natively supports a 262,144-token context window and can scale to one million tokens using YaRN. The technical report is on GitHub, and weights are available on Hugging Face and ModelScope. The production version ships as Qwen3.8-Flash through QwenCloud, priced at $0.16 per million input tokens and $0.47 per million output tokens. The API should go live shortly, according to Qwen.
