If you've ever tried to run a large language model on your own hardware, you've probably hit the same wall: the model is huge, your GPU's VRAM is not, and suddenly a 7B parameter model that "should" fit doesn't. This is where quantization comes in — and it's one of the most impactful techniques for making LLMs actually usable outside of a data center.
This post breaks down what quantization actually does, the major approaches you'll run into, and how to think about the trade-offs when picking one for your project.
The Core Idea
Every parameter in a neural network is a number, and by default those numbers are usually stored as 32-bit or 16-bit floating point values (FP32 or FP16/BF16). Quantization is the process of representing those same numbers with fewer bits — commonly 8-bit, 4-bit, or even lower.
Fewer bits per parameter means:
