Quantization

Definition

Quantization is storing a model's numbers in fewer bits. In practice that means one thing: Llama 3.1 70B needs about 141 GB of GPU memory for its weights in BF16, and about 35 GB in INT4. Same model, a quarter of the footprint — the difference between a two-GPU server and a single desktop-class card.

The trade is precision, and the interesting part is that the cost does not show up where you look for it. In a NeurIPS 2024 study, six different quantization schemes all scored within 0–2% of the 16-bit original on standard benchmarks — while changing up to 13.6% of the individual answers, correct becoming incorrect and incorrect becoming correct in roughly equal measure. The average survived. The model underneath it did not.

How It Works

A weight is just a number. What quantization changes is the format those numbers are stored in — how many bits, and how the bits are divided up.

The number-format ladder

A floating-point number splits its bits three ways: one sign bit, some exponent bits, and the rest on the mantissa (the significant digits). Exponent bits buy you range — how large and how small a value you can represent at all. Mantissa bits buy you precision — how finely you can distinguish two nearby values. Every format on this ladder is a different answer to how to split that budget.

Why BF16 beat FP16 and won training. FP16 spends 10 of its 16 bits on the mantissa and only 5 on the exponent, which caps it at 65,504 and — worse — makes small gradients underflow to zero during backpropagation. Teams worked around this with loss scaling: multiply the loss by a large constant to drag gradients back into range, then divide it out. BF16 removes the problem instead of managing it. It keeps FP32's 8 exponent bits, so its range is identical to FP32's, and pays for that out of the mantissa, which drops to 7 bits. Neural network training turns out to be far more sensitive to overflow and underflow than to rounding error, so this was the right trade — and BF16 has the pleasant property of being FP32 with the bottom 16 bits chopped off, which makes conversion nearly free.

TF32 is not a storage format at all. It is a compute mode: values still occupy 32 bits in memory, but the tensor core multiplies them with only 19 significant bits (8 exponent, 10 mantissa, 1 sign). You get FP32's range, roughly FP16's precision, and a large speedup on the matrix multiplication, for free, without changing a line of your model code.

INT8 and INT4 have no exponent. An integer format cannot represent a range; it represents evenly spaced steps. So the real value is reconstructed as scale × integer, where a group of weights shares one floating-point scale. All of the difficulty in low-bit quantization lives in choosing that scale, and in what happens when one number in the group is much bigger than the rest.

The memory ladder, in gigabytes

This is the calculation every practitioner actually runs. Llama 3.1 70B has 70,553,706,496 parameters (the model card's exact count — "70B" is a rounding). Multiply by bytes per parameter, counting 1 GB as 10⁹ bytes:

The BF16 row is the one worth staring at. The weights alone consume essentially the whole of an H200's advertised 141 GB, leaving nothing for activations or the KV cache — which is why a 70B model in BF16 is a two-GPU deployment in practice, and an FP8 one is not. That single step down the ladder halves the hardware bill.

It also makes the model faster, and for a reason that has nothing to do with arithmetic: token generation is limited by how quickly weights can be streamed out of GPU memory, so a step that moves half the bytes takes roughly half the time. That is the memory wall, and it is the whole reason quantization is a latency optimisation and not merely a storage one.

Calibration and the outlier problem

To pick a scale, you look at real data: run a few hundred sample inputs through the model, record how large the activations actually get, and set the scale so the largest of them just fits. This is calibration, and it is where naive quantization dies.

Transformers develop a small number of outlier features — activation dimensions whose magnitude is enormously larger than everything around them. Tim Dettmers' LLM.int8() work found these emerge across all layers at around 6.7B parameters and above, affecting 75% of hidden state sequences, while making up only about 0.1% of features. They are not noise: zeroing them out degrades validation perplexity by 600–1,000%.

One such value in a group drags the shared scale up, and every ordinary weight beside it collapses into a handful of quantization levels. This is why 8-bit quantization worked fine on small models and then broke at exactly the scale everyone cared about. Every method that works handles outliers specially rather than hoping they average out: LLM.int8() splits them into a separate 16-bit matrix multiply and keeps 99.9% of values in 8-bit; SmoothQuant shifts the difficulty off the activations, where outliers live, and onto the weights, where they are easier to handle; GPTQ and AWQ fit scales that minimise error on the calibration data; DeepSeek-V3 quantizes in 128-element tiles, so one outlier can only poison 128 numbers.

Types

There are exactly two, and the distinction is universal.

Post-training quantization (PTQ) takes a model that is already trained, runs a calibration set through it, computes scales, and writes out the low-bit weights. No gradients, no training data, no training infrastructure. It costs minutes to hours on a single GPU, which is why almost every quantized checkpoint you can download is a PTQ one — GPTQ, AWQ, SmoothQuant and bitsandbytes are all PTQ methods.

Quantization-aware training (QAT) simulates the rounding during a training or fine-tuning run, so the optimiser can see the damage and route around it: the model learns weights that happen to survive being crushed into 4 bits. It needs training data, training compute, and access to the pipeline — so in practice only the lab that made the model can do it well.

The rule of thumb: PTQ is good enough at 8 bits; QAT earns its cost at 4 bits and below. Google made the case concretely with Gemma 3, fine-tuning each model for roughly 5,000 QAT steps against the unquantized checkpoint's own probabilities, and reporting that this recovered as much as 54% of the quality lost to quantization. The 27B model went from 54 GB in BF16 to 14.1 GB in INT4 — small enough to run on a 24 GB consumer card.

Real-World Applications

Halving a serving bill. The FP8 row above is not a benchmark, it is a procurement decision: one H100 instead of two, for a model most teams cannot otherwise afford to self-host. The leftover memory is not spare — it is KV cache, and KV cache is batch size, so the same step buys throughput twice.

Local models exist because of INT4. The llama.cpp and Ollama ecosystem, and Google's decision to ship official QAT checkpoints, rest on a 4-bit 27B model fitting in 24 GB of consumer VRAM. Without quantization, "run a good model on your own machine" is not a product category.

FP8 has moved into training, not just inference. DeepSeek-V3 — 671B total parameters, 37B active — was trained with an FP8 mixed-precision framework: the compute-heavy matrix multiplies in FP8, the sensitive parts (embeddings, output head, MoE gating, normalisation, attention) deliberately held at higher precision. That is the first public validation of FP8 training at frontier scale.

Edge AI has no other option. The NPU in a phone is an INT8/INT4 engine. An FP32 model there is not slow — there is no path to run it at all.

Key Concepts

• Read the scheme name: W4A16 is 4-bit weights against 16-bit activations — memory shrinks, but the maths still runs in 16-bit, so you save bytes and not much else. W8A8 takes the activations down too, which is what actually unlocks the faster tensor-core path, and is also where the outlier problem bites hardest. Halving the weights and halving the latency are two different purchases.
• Granularity is the main dial: one scale per tensor is fastest and worst; per-channel is the common compromise; per-group (128 elements, say) is what modern 4-bit methods use, because a smaller group means an outlier can wreck fewer of its neighbours. Every step finer costs a little memory for the scales themselves.
• Quantization is not distillation or pruning: it keeps every parameter and shrinks each one. Distillation trains a smaller model to imitate a larger one; pruning deletes parameters outright. They compose — a distilled model can be quantized — but they fail in different ways.

Challenges

The benchmark holds still while the model moves. This is the failure mode that costs people real money, and it is invisible to the standard eval. In the NeurIPS study cited above, accuracy moved by 0–2% while up to 13.6% of answers flipped — the model was measurably different and statistically identical. If your only gate is a benchmark average, you will ship the change and find out from users.

The loss is not spread evenly; it concentrates in the tail. A 2024 study of quantized multilingual models found the drop was −0.7% for Latin-script languages against −1.9% for non-Latin scripts on a 103B model, and that mathematical reasoning on MGSM fell −13.1% on a 35B model under aggressive quantization. Most damningly: on Japanese, automatic metrics showed a −1.7% drop while human evaluators on realistic prompts reported −16.0%. Rare tokens, rare scripts and long chains of reasoning are exactly where the quantization error compounds — and exactly what your aggregate score cannot see.

The calibration set is a configuration option you probably did not know you set. Those same multilingual results used 128 English samples to calibrate. The scales a model gets are the scales its calibration data asks for, and if that data does not look like your traffic, the model is tuned for someone else's workload.

The KV cache is a separate budget. Quantizing weights does nothing for it, and at long context it can exceed the weights themselves — see KV cache; it can be quantized too, usually to 8-bit, with its own trade-offs.

What breaks if you get this wrong: you drop a model to INT4, MMLU is within a point, you ship — and three weeks later support is telling you the model got worse at the one thing your users actually do, and you have no eval that shows it. The fix is not more benchmarks. It is a paired comparison against the unquantized model on your own traffic: same prompts, both models, count the disagreements.

Future Trends

Four bits is becoming native. NVIDIA's Blackwell architecture executes NVFP4 directly on its tensor cores at roughly twice the throughput of FP8 — and the format is a direct answer to the outlier problem above: it scales in blocks of 16 elements, each with its own FP8 scale factor, so one large value can only distort fifteen neighbours. Each hardware generation promotes a precision that used to be a lossy compromise into a first-class format.

The second shift is that quantized checkpoints are becoming the vendor's job rather than the community's. Gemma 3's official QAT releases are the template: the lab that trained the model is the only party that can run QAT properly, and shipping a 4-bit checkpoint beside the BF16 one turns quantization from something done to a model into something the model was built for.