Why INT4 Weight-Only Quantization Doesn't Speed Up Prefill

You benchmark a 70B model with batch_size=1, one prompt, one stream. FP16 gives you 18 tokens/sec. You swap in an AWQ INT4 checkpoint and get 55 tokens/sec. Three times faster, same GPU, ~1 point of accuracy lost. You ship it.

Then production traffic arrives: 8k-token RAG prompts, 40 concurrent users. Time-to-first-token gets worse than the FP16 build, and your throughput at high concurrency is flat or slightly down. Nothing is broken. INT4 weight-only quantization did exactly what it does — it reduced bytes moved, and prefill was never bottlenecked on bytes.

TL;DR

INT4 weight-only quantization (W4A16) reduces memory traffic, not FLOPs. The kernel dequantizes INT4 back to FP16 and runs the same FP16 tensor-core MMA. Peak compute is unchanged.

Decode is memory-bandwidth-bound, so cutting weight bytes 4x gives close to a 4x speedup at low concurrency. Prefill is compute-bound, so it gets nothing — and pays the dequantization overhead.