Serving Gemma 4 E2B in pure JAX on AWS g5g.2xlarge and g6.2xlarge with a byte-identical payload. The older instance loses 87% of decode to dtype conversion, and nothing in the logs says so.
This article compares two AWS GPU instance families for serving a small language
model, using a payload that is byte-identical on both. The older family loses
87% of decode to dtype conversion, and nothing in any log, metric or health
check says so.
The code is here:
A field report on serving Gemma 4 E2B under vLLM on AWS G5g — the only aarch64 + SM 7.5 hardware there is. No published build…
This article provides a step by step deployment guide for serving Google's Gemma 4 on an AWS EC2 G5g...
One hand-written Gemma 4 port, no PyTorch and no vLLM, on two NVIDIA GPUs a generation apart. Most of it ports untouched. Two…
A Blog post by Nikhil K. on Hugging Face
A field report on running Google's Gemma-4 on AWS Inferentia2: mixed attention heads, the vLLM / optimum-neuron / NxD dead-ends,…
What it took to deploy, why the QAT checkpoints refuse to load, and what one flex-start v6e chip is actually worth — measured…
