Quantization is a mechanism for compressing a model by storing each weight in fewer bits. It reduces memory requirements and accelerates inference. But what happens beneath this saving? What exactly is lost, and why do models typically survive the compression?

Rounding to a smaller grid

At the heart of quantization sits a simple operation: rounding. At full precision, each weight can take one of tens of thousands of possible values. Quantization restricts these to a smaller grid — 256 values in the eight-bit case, or just 16 in the four-bit case. Each original weight is rounded to the nearest available grid point. A weight like 0.3847 might become 0.3861 — a small error, but a real one.

Error accumulation

This small error seems harmless in isolation. But a model has dozens of layers, and each layer’s error cascades into the next. This accumulation magnifies the initial rounding error as it propagates forward. Because of this compounding effect, a coarser quantization grid (fewer bits) degrades quality non-linearly; the loss accelerates.

The outlier problem

Quantization’s hardest challenge lies elsewhere: outliers. A model’s weight distribution is usually tightly clustered near zero. However, a few very large weights always exist. If you widen the quantization grid to accommodate these large outliers, the gap between grid points grows. Consequently, you lose precision near zero — exactly where the vast majority of weights live. Crude quantization does its worst damage here, which is why advanced methods handle outliers separately.

Vector-space collapse

Under extreme compression (say, two or three bits), a more severe failure mode emerges: distinct values get mapped to a single quantized value. When weights that were previously different become identical, the model loses its ability to discriminate. Its information capacity effectively collapses. This mechanism explains why quality loss at very low bit widths is sudden and severe, rather than a gradual degradation.

Why models are robust

Despite these mechanics, language models are remarkably robust to moderate quantization. Published measurements show a consistent pattern: at eight bits, quality loss is usually under one percent; at four bits, using a good method, it commonly sits in the one-to-three-percent range. (These are figures reported across the community and model families — not a guaranteed constant; a given model on a given task can sit outside the range in either direction.) Models survive this because their weights are the product of a statistical process, not a rigid computation. Internal redundancy across millions of parameters allows the network to route around small errors. A model typically has more parameters than a task strictly requires, and this margin is what makes quantization viable.

Which precision to reach for

Understanding these mechanics points to a practical default rather than a single correct answer. The available options form a ladder, representing different points on the memory-versus-fidelity trade-off:

  • FP16 (full) — the reference. No quantization loss, maximum memory footprint, and highest cost. This is the baseline everything else is measured against.
  • INT8 — a sensible default: near-baseline quality requiring roughly half the memory. When fidelity matters and the budget permits, this is the safe choice.
  • INT4 (GPTQ/AWQ) — the cost-sensitive rung. It fits significantly larger models onto the same hardware, but only with a good method and only after verification on your specific task. This is where outlier handling and vector-space collapse begin to bite, and some models degrade well past the reported average.
  • GGUF — the format to reach for when serving on a CPU or edge hardware, rather than a datacentre GPU.

Precision is one lever; the serving engine is a separate one. An engine with continuous batching (such as vLLM) increases GPU utilisation independently of the bit width. The two decisions compound; they do not substitute for each other. Furthermore, the operational cost of pushing to four bits is real. You trade a margin of accuracy and take on a per-model verification burden in exchange for memory savings. It is worth doing when a model would not otherwise fit, or when infrastructure cost is the absolute dominant factor; it is rarely worth it if a slightly smaller model running at INT8 can perform the same job with cleaner predictability.

Putting it together

Quantization is a pragmatic trade-off: you buy memory and speed at the cost of a minor loss in precision. Success depends on understanding the mechanical limits — exactly how far a system can be compressed before error accumulation and vector-space collapse degrade its outputs. Ultimately, the final measure of success is always the model’s behaviour on your real task, not the underlying bit count.