Where tensor-parallel inference hits the NVLink wall 2026-05-31 · GPU / distributed...
On 4× H100, all-reduce peaks at 366 GB/s—77% of NVLink ceiling—but LLM decode is bottlenecked by small-message latency, not throughput. Scaling TP past optimal slows per-token decode; measure small-message latency on the actual fabric before adding GPUs.
Where tensor-parallel inference hits the NVLink wall
2026-05-31 · GPU / distributed systems
Tensor parallelism splits each layer across GPUs, so every forward pass pays for an
all-reduce over the network fabric. On a single node that fabric is NVLink/NVSwitch — and
how close you get to its theoretical budget decides whether TP helps or hurts. This post
ThunderKittens Now Optimized for NVIDIA Blackwell GPUs
The First Disaggregated Inference Demo for AI Agents Is Live
NVIDIA Technical Blog
NeurIPS 2022: Overcoming communication bottlenecks for decentralized training (1/2)
FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling
DynoSim: Simulating the Pareto Frontier | NVIDIA Technical Blog
TPUs and Trainium beat NVIDIA on cost-per-token and watts-per-token. So why does almost everyone still run on NVIDIA? The answer…
Deploying large language models (LLMs) requires large-scale distributed inference, which spreads model computation and request…
NVIDIA GB200 NVL72 introduces a fundamentally new way to build GPU clusters by extending NVIDIA NVLink coherence across an entire…
In-depth technical review of Google's TPU Developer Hub: where it shines, where it hurts, real trade-offs, and how to decide…
Today we are announcing immediate access to Together GPU Clusters accelerated by the NVIDIA Blackwell platform, and an…
At Together AI, we have been investing in the ThunderKittens framework - a software library that we developed in collaboration…
