What Nvidia's first Groq 3 LPU benchmarks do and don't tell us about its $20B gamble

Nvidia’s $20 billion bet on Groq’s LPU tech sure looks like it was a good one. On Monday, the GPU giant offered the first glimpse of just how big a speedup its Groq 3-based LPX racks will provide.In an independent benchmark conducted by Artificial Analysis, Nvidia’s LPX rack systems managed to churn out 3,400 tokens a second (tok/s) with a 100,000-token input sequence in Google’s Gemma 4 31B model.According to Nvidia, this makes it 4x faster than the nearest alternative platform, which going off Artificial Analysis’ leaderboard would be a direct dig at Cerebras, which managed a still impressive 882 tok/s under the same conditions.

Acquihired by Nvidia in late December, Groq has LPUs that feature an SRAM-heavy dataflow architecture designed specifically for high-performance inference serving. Unlike traditional datacenter GPUs, which rely on high-speed DRAM memory tech like GDDR7 and HBM4, Groq’s chips rely entirely on a large pool of on-die SRAM orders of magnitude faster than even the best HBM stacks (around 2.75 TB/s each) available today. And when it comes to inference, memory bandwidth is the bottleneck to beat.

The third generation of the chips launched as part of Nvidia’s broader Vera Rubin platform earlier this year boasted 150 TB/s of memory bandwidth. The catch, of course, is that SRAM chews up a ton of die area, which means you can’t pack all that much into a reticle-sized part. Where Nvidia’s top-specced Rubin GPU boasts 288 GB of onboard memory, each Groq 3 LPU has just 500 MB — 576x less.That’s far too little memory to run Gemma 4 31B on a single LPU, so Nvidia’s architecture uses Ethernet to distribute models across multiple accelerators. Each LPX rack can be equipped with up to 256 LPUs for 128 GB of high bandwidth SRAM. For large models, multiple LPX racks can be ganged together. So why would anyone need to run Gemma 4 31B at 3,400 tokens a second? You probably don’t, but the AI code assistant or agent you're using it with might. The faster you can generate tokens, the longer models can reason, the more turns the agent can take, and the more information it can process or actions it can take in the same window of time.The big bet here is that, in the agentic age, faster inference servers make for smarter agents and are therefore worth a premium. And the combination of Nvidia GPUs and Groq 3 LPUs certainly seems to have caught Nvidia’s customers’ attention. On Monday, the GPU giant revealed that Netherlands-based neocloud Nebius would be among the first to field the combined systems in its datacenters.Just how impressive is Nvidia’s LPX system really?While 3,400 tok/s is impressive at face value, Gemma 4 31B is arguably a best case scenario for the hardware, and it remains to be seen how gracefully the architecture will scale to larger, more complex MoE models.At 31 billion parameters, the model fits neatly into a single LPX rack regardless of what data type Nvidia is using to store the weights. Nvidia is running the model at FP8, which means it needs a little over 31 GB or just under 64 LPUs of SRAM capacity.We’ve reached out to Nvidia for comment on how exactly it's distributing the model across those chips, but didn’t get a clear answer. Our best guess is plain old pipeline parallelism with possibly a dash of data parallelism to improve performance at higher concurrency levels.