Training LLMs at massive scale brings unique infrastructure challenges, especially as jobs span thousands of GPUs and run for extended periods. The longer these jobs run, the greater the likelihood of encountering unscheduled interruptions or resource fluctuations. Even infrequent device unavailability can have outsized effects on tightly interconnected clusters, resulting in slowdowns for a given training run.
For large-scale training jobs, elastically adapting the job to the number of available GPUs is a powerful method to improve Goodput. In the context of AI training, Goodput represents the critical measure of useful, convergence-driving work completed, rather than just raw hardware throughput.
Effective methods for elastic scaling today include dropping a data replica, utilizing fast checkpoint-restarts, or swapping to hot spares. These methods enable LLM jobs to adapt to GPU availability changes while maintaining balance across the system. However, they also incur some amount of lost throughput and higher cost during the period that the training job is running in a reduced availability condition.
A recent paper on Nonuniform Tensor Parallelism (NTP) introduces a forward-looking, experimental framework that builds on these methods in a way that minimizes throughput overheads. Combined with potential dynamic power boosting to offset any performance loss, throughput remains steady, transforming interruptions into manageable and recoverable events.
