I implemented GPTQ from scratch on a nanoGPT model and got only 1.1% perplexity degradation across 61 quantized layers. Here's exactly how it works and what I built.
1. The Problem with Naive Quantization
Quantization is one of the simplest and most effective ways to reduce the cost of running neural networks. Instead of storing weights in 32-bit floating point format, we reduce them to lower precision like INT8 or INT4. This reduces memory usage and can significantly speed up inference on hardware that supports low-precision arithmetic.
The simplest approach is Post-Training Quantization (PTQ), where each weight is independently rounded to the nearest quantized value. While this is fast and easy to implement, it ignores an important fact: neural network weights are not independent. Each weight contributes to a shared output, and small perturbations in one weight can interact with others in non-trivial ways.
Because of this, naive PTQ often introduces noticeable accuracy degradation. Some layers are extremely sensitive, and uniform rounding treats all weights as equally important. In practice, this leads to compounding errors across layers, especially in transformers where representations are tightly coupled.
