LLMs Don't Have to Generate One Token at a Time: How Medusa and Multi-Token Prediction Cheat Autoregression

Hello, I'm Shrijith Venkatramana, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.

A modern LLM can contain hundreds of billions of parameters, run on extremely expensive accelerators, and still spend most of its inference time doing something that looks embarrassingly sequential:

token 1 -> token 2 -> token 3 -> token 4 -> token 5 -> ...

Enter fullscreen mode

Exit fullscreen mode

LLMs Don't Have to Generate One Token at a Time: How Medusa and Multi-Token Prediction Cheat Autoregression

Related reading

Speculative Decoding: How LLMs Generate Tokens Faster Without Changing the…

Constitutional Methods for LLMs: Turning Written Principles into Training…

Multimodal Transformers: How LLMs Learn to See

BPE-Style Tokenizers: The Small Algorithm That Decides What an LLM Can See

Attention Mathematics: Encoder-Only vs Decoder-Only vs Encoder-Decoder LLMs

LLM Sampling, Demystified: Temperature, Top-k, Top-p, Min-p and Repetition…