Originally published on software-engineer-blog.com.
Your model works. It's just too big to ship. You have two standard paths to shrink it — and they are nothing alike.
Mental model: Quantization is compression; distillation is re-education. One shrinks the same brain, the other builds a smaller one that imitates the original.
Quantization: Same Brain, Fewer Bits
Quantization takes your trained model and re-encodes its weights at lower precision. Instead of storing each weight as a 32-bit float (FP32), you store it as a 16-bit half (FP16), an 8-bit integer (INT8), or even a 4-bit integer (INT4).
