LLM Sampling, Demystified: Temperature, Top-k, Top-p, Min-p and Repetition Penalty

Hello, I'm Shrijith Venkatramana, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.

A language model does not actually "write" the next token.

It produces a probability distribution over the vocabulary and says, roughly:

Here are 50,000 things I could say next. These are how plausible I think they are.

Everything that happens after that is decoding.

LLM Sampling, Demystified: Temperature, Top-k, Top-p, Min-p and Repetition Penalty

Related reading

BPE-Style Tokenizers: The Small Algorithm That Decides What an LLM Can See

Multimodal Transformers: How LLMs Learn to See

LLMs Don't Have to Generate One Token at a Time: How Medusa and Multi-Token…

Loss Landscapes of LLMs: The Map Beneath Gradient Descent

Constitutional Methods for LLMs: Turning Written Principles into Training…

Tree of Thoughts and MCTS for LLMs: What Happens When You Stop Making the Model…