A seventy-billion-parameter model needs about 140 GB of memory at FP16 precision — an allocation that typically dictates eight 80 GB A100 cards. For most teams, that infrastructure is out of reach. With a technique called quantization, you can compress that same model until it fits on a single GPU. Here, we walk through the memory math step by step to examine what you gain and what you give up.
The memory math
The baseline calculation is straightforward: the number of parameters multiplied by bytes per parameter. At FP16, each parameter requires two bytes, so seventy billion parameters consume roughly 140 GB. Quantization reduces the bits allocated per parameter:
| Precision | Bits per parameter | Memory for a 70B model |
|---|---|---|
| FP16 | 16 | ~140 GB |
| INT8 | 8 | ~70 GB |
| INT4 | 4 | ~35 GB |
At INT8, the model footprint halves, fitting cleanly onto a single 80 GB card. At INT4, it drops to roughly 35 GB — small enough for larger consumer hardware, leaving capacity for the KV cache and batch processing.
What quantization actually does
Quantization rounds each weight to the nearest value on a constrained numerical grid. FP16 provides more than sixty thousand possible values; INT8 offers 256, and INT4 holds just 16. This rounding introduces error. However, language models are fundamentally robust to this degradation. Their scale and internal redundancy allow them to “route around” small errors.
What you give up
The trade-off for quantization is a quality drop that scales with how aggressively you compress:
- INT8: under one percent quality loss, and usually about twice as fast. In practice, the difference is rarely noticeable.
- INT4: roughly one to three percent quality loss, provided you use an optimised method. This remains acceptable for most uses.
- Below INT4 (e.g. INT2): the loss grows sharply; this range isn’t recommended for serious work.
The underlying method determines the severity of this drop. Naive quantization can degrade quality by five to ten percent, while more sophisticated approaches achieve identical compression with minimal penalty.
The outlier problem
Why do crude methods fail? The primary reason is outliers. Among a model’s weights, a few carry very large values. If you stretch the quantization grid to accommodate those extremes, you sacrifice precision across the narrower range where the vast majority of weights reside. Advanced methods are designed specifically to resolve this imbalance.
The practical recipe
For INT4 quantization on a large model, GPTQ and AWQ are the standard choices. Both manage outliers effectively, keeping quality loss minimal. The NF4 format, used in QLoRA, is similarly optimised for typical weight distributions. The practical sequence is straightforward: try INT8 first. If the model fits your hardware and maintains quality, stop there. If you require tighter compression, move to INT4 using a proven method, and measure the output against your actual tasks rather than public benchmarks.
Putting it together
Quantization alters the operational cost equation. Workloads that required eight GPUs previously can now run on one. Success depends on selecting the correct compression threshold — compressing enough to fit the target hardware, not so much that quality collapses. And as always, the final judge isn’t the number on paper but how the model behaves on your real data.