Adam and AdamW: The Optimizer That Made Modern LLM Training Possible

Hello, I'm Shrijith Venkatramana, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.

Most people learn neural networks by staring at the model.

Weights. Attention. MLPs. LayerNorm. Tokenizers. Context windows.

But when you actually train an LLM, there is another piece of machinery making billions of decisions every second:

the optimizer.

Adam and AdamW: The Optimizer That Made Modern LLM Training Possible

Related reading

Adam: The Optimization Algorithm That Made LLMs Practical

Reward Hacking in LLMs: When the Model Learns to Win the Game Instead of Doing…

Generative Pre-Training and Discriminative Fine-Tuning: The Two-Step Recipe…

KV Cache in LLMs: The Optimization That Makes Modern AI Models Feel Fast

Attention Mechanisms in LLMs: The Idea That Changed AI Forever

FlashAttention Explained: The Optimization That Made Modern LLMs Practical