If you have implemented a transformer model in PyTorch, you can use the same code for both training and inference, but in very different ways. During training, you usually process a batch of fixed-length token sequences and update the model weights. During inference, the weights are fixed and the model generates new tokens one at a time.
This difference changes almost everything about performance. Training is dominated by large matrix multiplications and the backward pass. Inference is dominated by repeated forward passes, memory movement, and the need to keep previous attention keys and values available for the next token.
In this chapter, you will learn about:
The autoregressive generation loop
The difference between prefill and decode
