In the last entry I got Gemma-4's 128-expert MoE running on an inf2.24xlarge and signed off with...
In the last entry I got Gemma-4's 128-expert MoE running on an inf2.24xlarge and signed off with a
cliffhanger: fitting it on a 2-core box "needs fp4 — a separate expedition." This is that expedition.
It ended nowhere near where I thought it would: not with fp4, and not on the 8xlarge I was
aiming for, but on the smallest, cheapest Inferentia2 instance AWS sells — a single inf2.xlarge
with 16 GB of host RAM — running a 26B-parameter model. Here's the refinement trail, dead ends
In the last entry I got Gemma-4's 128-expert MoE running on an inf2.24xlarge and signed off with...
The MoE was the hard one: a dual-path FFN, a sparse expert loop that won't trace, and a bug where the device output was empty…
A side-by-side of all five Gemma-4 variants on Inferentia2 — PLE, KV-sharing, MatFormer, mixed attention, and a 128-expert MoE —…
Porting Gemma-4 31B (dense) to AWS Inferentia2: the tensor-parallel recipe that worked at 12B collapses at 31B, NxD ModelBuilder…
A field report on running Google's Gemma-4 on AWS Inferentia2: mixed attention heads, the vLLM / optimum-neuron / NxD dead-ends,…
A Blog post by Nikhil K. on Hugging Face
