A language model does not write text directly. Instead, it returns logits for the next token. The decoding algorithm decides how to turn those logits into a token, and repeating this decision produces the output text.
The decoding algorithm affects the behavior of the model. Greedy decoding is deterministic and stable, but it can be dull. Sampling introduces some randomness, which can produce more diverse text but may also produce mistakes. Beam search can be useful for some constrained tasks but is usually not the best default for chat-style generation. Output constraints can make the model produce JSON or stop at a specific marker.
In this chapter, you will learn about:
Greedy decoding
Temperature sampling
