Two AI computing clusters built from identical NVIDIA H100, GB200 NVL72, or GB300 NVL72 systems can deliver materially different training throughput. We routinely see 8% to 12% gaps between partner deployments and the corresponding NVIDIA reference architecture (RA) on the same workload, same model, same global batch size.
The cause is often a stack of configuration choices in the kernel, hypervisor, BIOS, and NVIDIA Collective Communications Library (NCCL) settings, each costing a few percent, that compound into a gap large enough to miss the 95% threshold required for NVIDIA Exemplar Cloud validation.
This post walks through four debugging investigations from real partner clusters. Each diagnostic isolates a distinct layer of the stack: system memory management unit (SMMU) and page-table behavior on NVIDIA Grace CPU; power management and non-uniform memory access (NUMA) placement on x86-based CPU; NVIDIA NCCL queue-pair concurrency on 1.6 Tbps fabrics; and silent hardware-installation defects. The post also shows the specific signal in perf, NVIDIA Nsight Systems, or NVIDIA NCCL tests that pointed to the root cause, alongside the tuning change that closed the gap.
Infrastructure engineers and performance architects who already run these benchmarks can benefit from these diagnostic patterns we use internally to run against their own clusters before formal RA validation.
