Porting Gemma-4 31B (dense) to AWS Inferentia2: the tensor-parallel recipe that worked at 12B collapses at 31B, NxD ModelBuilder saves it — and then a passing validation lies to your face.
I ported Google's biggest open Gemma-4 — the 30-billion-parameter dense model — to AWS Inferentia2.
The compiled model passed my strictest check: its output was token-for-token identical to the
CPU reference. It was also complete gibberish. Both facts were true at the same time, and the reason
is a lesson worth more than the port.
This is the sequel to porting the smaller Gemma-4 models (E2B/E4B/12B). Those were hard because of
A field report on running Google's Gemma-4 on AWS Inferentia2: mixed attention heads, the vLLM / optimum-neuron / NxD dead-ends,…
A side-by-side of all five Gemma-4 variants on Inferentia2 — PLE, KV-sharing, MatFormer, mixed attention, and a 128-expert MoE —…
The 12B ships as a multimodal class with no encoder loaded, and its sliding-window attention overflows Neuron's fused-attention…
The MoE was the hard one: a dual-path FFN, a sparse expert loop that won't trace, and a bug where the device output was empty…
In the last entry I got Gemma-4's 128-expert MoE running on an inf2.24xlarge and signed off with...
How to get Google's Gemma 4 26B-A4B Mixture-of-Experts model running locally — including speculative...
