---
source: 'https://howaiworks.ai/glossary/quantization'
section: glossary
title: Quantization
description: >-
  Running a model in fewer bits — FP16, FP8, INT4 — to cut memory and cost, what
  accuracy it actually costs, and why a memory-bound model also gets faster.
tags:
  - Quantization
  - Inference
  - Efficiency
  - Model Compression
  - GPU
  - Edge AI
  - Memory Bandwidth
  - Deep Learning
category: Deep Learning
datePublished: '2026-07-13'
lastUpdated: '2026-07-24'
---

# Quantization

> Running a model in fewer bits — FP16, FP8, INT4 — to cut memory and cost, what accuracy it actually costs, and why a memory-bound model also gets faster.

## Definition

**Quantization is storing a model's numbers in fewer bits.** In practice that means one thing:
[Llama 3.1 70B](https://howaiworks.ai/models/llama) needs about **141 GB** of GPU memory for its
[weights](https://howaiworks.ai/glossary/weights) in BF16, and about **35 GB** in INT4. Same model, a quarter of the
footprint — the difference between a two-GPU server and a single desktop-class card.

The trade is precision, and the interesting part is that the cost does not show up where you look
for it. Quantization is the clearest case of the general shape described under
[performance](https://howaiworks.ai/glossary/performance) — quality, speed and cost as one frontier rather than three
independent numbers, where buying on one axis is always paid for on another. In a NeurIPS 2024 study, six different quantization schemes all scored within **0–2%** of
the 16-bit original on standard benchmarks — while changing up to **13.6%** of the individual
answers, correct becoming incorrect and incorrect becoming correct in roughly equal measure. The
average survived. The model underneath it did not.

## How It Works

A weight is just a number. What quantization changes is the *format* those numbers are stored in —
how many bits, and how the bits are divided up.

### The number-format ladder

A floating-point number splits its bits three ways: one **sign** bit, some **exponent** bits, and
the rest on the **mantissa** (the significant digits). Exponent bits buy you *range* — how large
and how small a value you can represent at all. Mantissa bits buy you *precision* — how finely you
can distinguish two nearby values. Every format on this ladder is a different answer to how to
split that budget.

| Format | Bits | Sign / Exp / Mantissa | Max magnitude | Where it is used |
|---|---|---|---|---|
| FP32 | 32 | 1 / 8 / 23 | ~3.4 × 10³⁸ | The default everywhere else in computing |
| TF32 | 19 (in a 32-bit slot) | 1 / 8 / 10 | ~3.4 × 10³⁸ | NVIDIA tensor cores, from Ampere on |
| BF16 | 16 | 1 / 8 / 7 | ~3.4 × 10³⁸ | The standard for training |
| FP16 | 16 | 1 / 5 / 10 | 65,504 | Older mixed-precision training; inference |
| FP8 (E4M3) | 8 | 1 / 4 / 3 | 448 | Weights and activations |
| FP8 (E5M2) | 8 | 1 / 5 / 2 | 57,344 | Gradients, which need range over precision |
| INT8 / INT4 | 8 / 4 | integer + shared scale | set by the scale | Inference, edge devices |

**Why BF16 beat FP16 and won training.** FP16 spends 10 of its 16 bits on the mantissa and only 5
on the exponent, which caps it at 65,504 and — worse — makes small gradients underflow to zero
during backpropagation. Teams worked around this with *loss scaling*: multiply the loss by a large
constant to drag gradients back into range, then divide it out. BF16 removes the problem instead of
managing it. It keeps FP32's **8 exponent bits**, so its range is identical to FP32's, and pays for
that out of the mantissa, which drops to 7 bits. Neural network training turns out to be far more
sensitive to overflow and underflow than to rounding error, so this was the right trade — and BF16
has the pleasant property of being FP32 with the bottom 16 bits chopped off, which makes conversion
nearly free.

**TF32 is not a storage format at all.** It is a *compute* mode: values still occupy 32 bits in
memory, but the tensor core multiplies them with only 19 significant bits (8 exponent, 10 mantissa,
1 sign). You get FP32's range, roughly FP16's precision, and a large speedup on the
[matrix multiplication](https://howaiworks.ai/glossary/matrix-multiplication), for free, without changing a line of your
model code.

**INT8 and INT4 have no exponent.** An integer format cannot represent a range; it represents
evenly spaced steps. So the real value is reconstructed as `scale × integer`, where a group of
weights shares one floating-point `scale`. All of the difficulty in low-bit quantization lives in
choosing that scale, and in what happens when one number in the group is much bigger than the rest.

### The memory ladder, in gigabytes

This is the calculation every practitioner actually runs. Llama 3.1 70B has **70,553,706,496**
parameters (the model card's exact count — "70B" is a rounding). Multiply by bytes per parameter,
counting 1 GB as 10⁹ bytes:

| Precision | Bytes/param | Weight memory | What it fits on |
|---|---|---|---|
| FP32 | 4 | **282 GB** | 4 × H100 80GB (320 GB) — nobody does this |
| BF16 / FP16 | 2 | **141 GB** | 2 × H100 80GB. A single H200 has 141 GB — a dead heat |
| FP8 | 1 | **70.6 GB** | 1 × H100 80GB, with ~9 GB left over |
| INT4 | 0.5 | **35.3 GB** | 1 × L40S (48 GB), or 2 × RTX 4090 (24 GB each) |

The BF16 row is the one worth staring at. The weights alone consume essentially the whole of an
H200's advertised 141 GB, leaving nothing for activations or the
[KV cache](https://howaiworks.ai/glossary/kv-cache) — which is why a 70B model in BF16 is a two-GPU deployment in
practice, and an FP8 one is not. That single step down the ladder halves the hardware bill.

It also makes the model *faster*, and for a reason that has nothing to do with arithmetic: token
generation is limited by how quickly weights can be streamed out of GPU memory, so a step that
moves half the bytes takes roughly half the time. That is the [memory wall](https://howaiworks.ai/glossary/memory-wall),
and it is the whole reason quantization is a latency optimisation and not merely a storage one.

### Calibration and the outlier problem

To pick a scale, you look at real data: run a few hundred sample inputs through the model, record
how large the activations actually get, and set the scale so the largest of them just fits. This is
**calibration**, and it is where naive quantization dies.

Transformers develop a small number of **outlier features** — activation dimensions whose magnitude
is enormously larger than everything around them. Tim Dettmers' LLM.int8() work found these emerge
across all layers at around **6.7B parameters** and above, affecting 75% of hidden state sequences,
while making up only about **0.1%** of features. They are not noise: zeroing them out degrades
validation perplexity by **600–1,000%**.

One such value in a group drags the shared scale up, and every ordinary weight beside it collapses
into a handful of quantization levels. This is why 8-bit quantization worked fine on small models
and then broke at exactly the scale everyone cared about. Every method that works handles outliers
specially rather than hoping they average out: LLM.int8() splits them into a separate 16-bit matrix
multiply and keeps 99.9% of values in 8-bit; SmoothQuant shifts the difficulty off the activations,
where outliers live, and onto the weights, where they are easier to handle; GPTQ and AWQ fit scales
that minimise error on the calibration data; DeepSeek-V3 quantizes in 128-element tiles, so one
outlier can only poison 128 numbers.

## Types

There are exactly two, and the distinction is universal.

**Post-training quantization (PTQ)** takes a model that is already trained, runs a calibration set
through it, computes scales, and writes out the low-bit weights. No gradients, no training data,
no training infrastructure. It costs minutes to hours on a single GPU, which is why almost every
quantized checkpoint you can download is a PTQ one — GPTQ, AWQ, SmoothQuant and bitsandbytes are
all PTQ methods.

**Quantization-aware training (QAT)** simulates the rounding *during* a training or fine-tuning
run, so the optimiser can see the damage and route around it: the model learns weights that happen
to survive being crushed into 4 bits. It needs training data, training compute, and access to the
pipeline — so in practice only the lab that made the model can do it well.

The rule of thumb: **PTQ is good enough at 8 bits; QAT earns its cost at 4 bits and below.** Google
made the case concretely with [Gemma 3](https://howaiworks.ai/models/gemma), fine-tuning each model for roughly 5,000
QAT steps against the unquantized checkpoint's own probabilities, and reporting that this recovered
as much as 54% of the quality lost to quantization. The 27B model went from **54 GB in BF16 to
14.1 GB in INT4** — small enough to run on a 24 GB consumer card.

## Real-World Applications

**Halving a serving bill.** The FP8 row above is not a benchmark, it is a procurement decision: one
H100 instead of two, for a model most teams cannot otherwise afford to self-host. The leftover
memory is not spare — it is KV cache, and KV cache is batch size, so the same step buys throughput
twice.

**Local models exist because of INT4.** The llama.cpp and Ollama ecosystem, and Google's decision
to ship official QAT checkpoints, rest on a 4-bit 27B model fitting in 24 GB of consumer VRAM.
Without quantization, "run a good model on your own machine" is not a product category.

**FP8 has moved into training, not just [inference](https://howaiworks.ai/glossary/inference).** DeepSeek-V3 — 671B
total parameters, 37B active — was trained with an FP8 mixed-precision framework: the compute-heavy
matrix multiplies in FP8, the sensitive parts (embeddings, output head,
[MoE](https://howaiworks.ai/glossary/mixture-of-experts) gating, normalisation, attention) deliberately held at higher
precision. That is the first public validation of FP8 training at frontier scale.

**[Edge AI](https://howaiworks.ai/glossary/edge-ai) has no other option.** The NPU in a phone is an INT8/INT4 engine.
An FP32 model there is not slow — there is no path to run it at all.

## Key Concepts

- **Read the scheme name**: `W4A16` is 4-bit weights against 16-bit activations — memory shrinks,
  but the maths still runs in 16-bit, so you save bytes and not much else. `W8A8` takes the
  activations down too, which is what actually unlocks the faster tensor-core path, and is also
  where the outlier problem bites hardest. Halving the weights and halving the latency are two
  different purchases.
- **Granularity is the main dial**: one scale per tensor is fastest and worst; per-channel is the
  common compromise; per-group (128 elements, say) is what modern 4-bit methods use, because a
  smaller group means an outlier can wreck fewer of its neighbours. Every step finer costs a little
  memory for the scales themselves.
- **Quantization is not [distillation](https://howaiworks.ai/glossary/knowledge-distillation) or pruning**: it keeps
  every parameter and shrinks each one. Distillation trains a smaller model to imitate a larger
  one; pruning deletes parameters outright. They compose — a distilled model can be quantized — but
  they fail in different ways.

## Challenges

**The benchmark holds still while the model moves.** This is the failure mode that costs people
real money, and it is invisible to the standard eval. In the NeurIPS study cited above, accuracy
moved by 0–2% while up to 13.6% of answers flipped — the model was measurably *different* and
statistically *identical*. If your only gate is a benchmark average, you will ship the change and
find out from users.

**The loss is not spread evenly; it concentrates in the tail.** A 2024 study of quantized
multilingual models found the drop was **−0.7% for Latin-script languages against −1.9% for
non-Latin scripts** on a 103B model, and that mathematical reasoning on MGSM fell **−13.1%** on a
35B model under aggressive quantization. Most damningly: on Japanese, automatic metrics showed a
**−1.7%** drop while human evaluators on realistic prompts reported **−16.0%**. Rare tokens, rare
scripts and long chains of reasoning are exactly where the quantization error compounds — and
exactly what your aggregate score cannot see.

**The calibration set is a configuration option you probably did not know you set.** Those same
multilingual results used 128 *English* samples to calibrate. The scales a model gets are the
scales its calibration data asks for, and if that data does not look like your traffic, the model
is tuned for someone else's workload.

**The KV cache is a separate budget.** Quantizing weights does nothing for it, and at long context
it can exceed the weights themselves — see [KV cache](https://howaiworks.ai/glossary/kv-cache); it can be quantized too,
usually to 8-bit, with its own trade-offs.

**What breaks if you get this wrong:** you drop a model to INT4, MMLU is within a point, you ship —
and three weeks later support is telling you the model got worse at the one thing your users
actually do, and you have no eval that shows it. The fix is not more benchmarks. It is a *paired*
comparison against the unquantized model on your own traffic: same prompts, both models, count the
disagreements.

## Future Trends

Four bits is becoming native. NVIDIA's Blackwell architecture executes **NVFP4** directly on its
tensor cores at roughly twice the throughput of FP8 — and the format is a direct answer to the
outlier problem above: it scales in blocks of 16 elements, each with its own FP8 scale factor, so
one large value can only distort fifteen neighbours. Each hardware generation promotes a precision
that used to be a lossy compromise into a first-class format.

The second shift is that quantized checkpoints are becoming the vendor's job rather than the
community's. Gemma 3's official QAT releases are the template: the lab that trained the model is
the only party that can run QAT properly, and shipping a 4-bit checkpoint beside the BF16 one turns
quantization from something done *to* a model into something the model was built for.

## Frequently Asked Questions

### How much accuracy does quantization cost?

It depends on the workload, and the honest answer is that a benchmark score will not tell you. In a NeurIPS 2024 study, six quantization schemes all landed within 0–2% of the 16-bit model's accuracy, yet up to 13.6% of individual answers flipped between correct and incorrect. The average held; the behaviour changed.

### What is bfloat16 and why is it used instead of float16?

BF16 spends 1 bit on sign, 8 on the exponent and 7 on the mantissa. That gives it the same numeric range as FP32 with half the storage — at the cost of precision. FP16 has only 5 exponent bits, so it overflows above 65,504 and small gradients underflow to zero, which is why FP16 training needs loss scaling and BF16 does not.

### What is FP8?

An 8-bit floating-point format standardised by NVIDIA, Arm and Intel in two variants: E4M3 (4 exponent bits, 3 mantissa bits, max ±448), used for weights and activations, and E5M2 (5 exponent, 2 mantissa, max ±57,344), used for gradients, which need more range than precision.

### What is the difference between PTQ and QAT?

Post-training quantization (PTQ) converts an already-trained model using a small calibration set and no gradient updates — cheap, and usually good enough at 8-bit. Quantization-aware training (QAT) simulates the rounding during a training or fine-tuning run so the model learns weights that survive it — expensive, and what you reach for at 4-bit and below.

### Why does a quantized model run faster, not just smaller?

Token-by-token generation is limited by how fast weights can be read out of GPU memory, not by arithmetic. Halving the bytes per weight roughly halves the time that read takes. This is the memory wall, and it is why FP8 speeds up a model that was never compute-bound in the first place.

### What is the outlier problem in quantization?

Integer quantization maps a group of values onto a shared scale set by the largest magnitude in the group. A handful of very large activations stretch that scale and crush everything else toward zero. These outlier features emerge reliably in transformers at around 6.7B parameters and above, which is why naive 8-bit quantization breaks at exactly the scale people care about.

## Related

### Related terms

- [Model Compression](https://howaiworks.ai/glossary/model-compression)
- [Memory Wall](https://howaiworks.ai/glossary/memory-wall)
- [Inference](https://howaiworks.ai/glossary/inference)
- [Model Size](https://howaiworks.ai/glossary/model-size)
- [Parameters](https://howaiworks.ai/glossary/parameters)
- [Edge AI](https://howaiworks.ai/glossary/edge-ai)

---

Source: https://howaiworks.ai/glossary/quantization — HowAIWorks.ai