Self-hosting a lite agent backend on one TPU: Gemma 4 E2B + vLLM on a v5e-1

Self-hosting a lite agent backend on one TPU chip

A single Google Cloud TPU v5e chip — 16 GB of HBM, about $0.58/hour on spot — will serve google/gemma-4-E2B-it under vLLM at 1,496 output tokens/sec aggregate, with 8.02 ms per-token latency at single stream and native tool-calling. That is enough to back a fleet of 8–16 concurrent "lite" agents for roughly $0.107 per million output tokens.

This is a build log with numbers. Everything here was measured on the hardware, and the sections that say "I was wrong about this" are the ones worth your time — four of my confident predictions were falsified by the benchmark, and each falsification was more useful than the guess.

Setup under test: v5litepod-1 (one v5e chip), us-west4-a, vllm/vllm-tpu:nightly, vLLM 0.26.1rc1.dev125+ga7a204cc6, tpu-inference JAX backend, google/gemma-4-E2B-it at bf16.

Part 1 — Scaffold and run