Advanced GPU Optimization: How to tech an LLM with CUDA and ROCm? - Part 4

Welcome to the grand finale! In Part 1, we learned how to walk (GEMM). In Part 2, we learned how to run (backprop & AdamW). In Part 3, we learned how to fly across thousands of GPUs (distributed training). But if you are still here, you aren't satisfied with just "working" code. You want perfection. You want to squeeze every last drop of FLOPS out of your H100 or MI300X.

In this fourth and final part, we stop treating the GPU as a generic processor and start treating it as a memory-bound machine. We will implement:

Flash Attention – The IO-aware kernel that made 100k+ context windows possible.

FP8 Quantization (Transformer Engine) – Using 8-bit floating point for 2x speedups.

Kernel Fusion – Fusing LayerNorm, residual adds, and dropout into a single pass.