How to Speed Up Transformer Training Using NVIDIA Apex (FusedAdam, FusedLayerNorm) and Native torch.amp

Benchmark NVIDIA Apex FusedAdam and FusedLayerNorm against PyTorch, then pair fused kernels with torch.amp for faster Transformer training.

Raccontata damarktechpost.com