A field report on running Google's Gemma-4 on AWS Inferentia2: mixed attention heads, the vLLM / optimum-neuron / NxD dead-ends, and the neuronx-cc compiler limits.
A field report on running Google's Gemma-4 family on AWS Inferentia2 (inf2), covering the
three architectural obstacles that break the vendor stack — **mixed attention heads, the
**vLLM / optimum-neuron / NxD* dead-ends, and the Neuron compiler (neuronx-cc) limits —
and the recipe that got all three model sizes serving coherently.*
Models
A side-by-side of all five Gemma-4 variants on Inferentia2 — PLE, KV-sharing, MatFormer, mixed attention, and a 128-expert MoE —…
Porting Gemma-4 31B (dense) to AWS Inferentia2: the tensor-parallel recipe that worked at 12B collapses at 31B, NxD ModelBuilder…
The 12B ships as a multimodal class with no encoder loaded, and its sliding-window attention overflows Neuron's fused-attention…
This is a submission for the Gemma 4 Challenge: Write About Gemma 4 Google released four Gemma 4...
Learn how to deploy Gemma 4 26B on legacy CPU hardware using quantization and optimization techniques for surprisingly effective…
A Blog post by Nikhil K. on Hugging Face
