I've been trying to understand what actually happens when a piece of ML code reaches the GPU. Not the usual model.cuda() and GPU go brrrr kind of understanding, but the actual stuff underneath it — threads, warps, memory, kernel launches, and why some operations are ridiculously fast while others suddenly become slow.
That rabbit hole eventually led me to Triton.
At first, I was wondering why anyone would even bother writing custom GPU kernels when PyTorch already gives us highly optimized operations. But the more I looked into it, the more I realized that GPU performance isn't always about how much computation you're doing. A lot of the time, it's about how you're moving data around.
CUDA gives you an insane amount of control over the GPU, but that control comes with a lot of complexity. You have to think about threads, warps, synchronization, memory access, and a bunch of other low-level details. That's great if you're comfortable with CUDA, but if you're coming from Python and PyTorch, it can be a pretty steep jump.
PyTorch makes this much easier through its higher-level abstractions. You can just write something like torch.softmax() and let the framework handle everything underneath. The problem is that when you're executing operations eagerly, individual operations can result in separate kernel launches and repeated reads and writes to global GPU memory. The code is easy to write, but there can be a lot of unnecessary movement of data happening underneath.
