The MoE was the hard one: a dual-path FFN, a sparse expert loop that won't trace, and a bug where the device output was empty while the CPU reference was perfect and every unit test passed.
Of the five Gemma-4 models I ported to AWS Inferentia2, the 26B-A4B was the one that was supposed to
be impossible: a Mixture of Experts. It compiled on the first try — then produced nothing. The
device output was empty. The CPU reference was perfect. My unit tests all passed. The bug was in a
place I'd have sworn couldn't have one.
This is the MoE entry in the series. The dense models (E2B/E4B/12B/31B) are hard because of
A side-by-side of all five Gemma-4 variants on Inferentia2 — PLE, KV-sharing, MatFormer, mixed attention, and a 128-expert MoE —…
Porting Gemma-4 31B (dense) to AWS Inferentia2: the tensor-parallel recipe that worked at 12B collapses at 31B, NxD ModelBuilder…
A field report on running Google's Gemma-4 on AWS Inferentia2: mixed attention heads, the vLLM / optimum-neuron / NxD dead-ends,…
In the last entry I got Gemma-4's 128-expert MoE running on an inf2.24xlarge and signed off with...
In the last entry I got Gemma-4's 128-expert MoE running on an inf2.24xlarge and signed off with...
This is a submission for the Gemma 4 Challenge: Write About Gemma 4 Gemma 4's most interesting model...
