---
source: 'https://howaiworks.ai/glossary/model-compression'
section: glossary
title: Model Compression
description: >-
  Making a trained model smaller and cheaper to run — quantization, pruning,
  distillation and low-rank methods, and which one your problem actually needs.
tags:
  - Model Compression
  - Quantization
  - Inference
  - Efficiency
  - Edge AI
  - GPU
  - Deep Learning
category: Deep Learning
datePublished: '2026-07-13'
lastUpdated: '2026-07-21'
---

# Model Compression

> Making a trained model smaller and cheaper to run — quantization, pruning, distillation and low-rank methods, and which one your problem actually needs.

## Definition

**Model compression is making an already-trained model cheaper to run, and choosing a method is
really choosing which resource you are short of.** There are four, and they do not do the same job:
[quantization](https://howaiworks.ai/glossary/quantization) cuts the *bytes per weight*, pruning cuts the *number of
weights*, [knowledge distillation](https://howaiworks.ai/glossary/knowledge-distillation) trains a *different, smaller
model* to imitate the one you have, and low-rank factorization replaces one big matrix with the
product of two small ones.

The distinction that matters before any of the technical detail is this: **quantization and pruning
keep your model, and distillation and low-rank factorization hand you a new one.** A new model means
new weights to ship, a training run to pay for, and an evaluation suite that no longer applies. Most
teams reach for the first pair because the second pair is expensive — and then discover that the
first pair cannot close the gap they have.

## How It Works

Every method removes something different, and that is the whole basis of the choice. Quantization
leaves all 70 billion [parameters](https://howaiworks.ai/glossary/parameters) in place and stores each in 4 bits instead
of 16. Pruning leaves the format alone and deletes parameters. Distillation deletes neither: it
trains a smaller network from scratch, supervised by the big one's outputs. Low-rank factorization
keeps the layer but re-expresses its weight matrix in fewer numbers.

| Method | What it removes | Cost to apply | Model you end up with |
|---|---|---|---|
| Quantization | Bits per weight | Hours on one GPU | The same model, re-encoded |
| Pruning | Weights | Cheap to prune, costly to recover | The same model, with holes |
| Distillation | Nothing — retrains | A training run and data | A different, smaller model |
| Low-rank factorization | Matrix rank | Hours, plus recovery | The same model, refactored |

They compose, and the order is not arbitrary: **prune, then retrain or distil to recover what you
broke, then quantize last.** Quantization is a final lossy encoding, and its calibration should see
the weights you are actually shipping — quantize first and you will re-damage the result when you
prune. This is old news, not a new idea: the 2015 *Deep Compression* paper ran exactly this pipeline
(pruning, then trained quantization, then Huffman coding) and shrank AlexNet **35x, from 240 MB to
6.9 MB, with no loss of accuracy**, and VGG-16 **49x**.

### The trap: sparsity your hardware cannot use

SparseGPT can prune OPT-175B to **50% sparsity in one shot with no retraining and negligible
perplexity change** — half the weights, gone, for free. It sounds like a two-times speedup. It is
usually a zero-times speedup, and this is the single most useful thing to know before you prune.

GPU sparse tensor cores accelerate exactly one pattern: **2:4**, meaning two of every four
consecutive weights must be zero, which is worth up to **2x throughput** on that matrix multiply.
An unstructured mask — the naturally best 50% of weights to delete, wherever they happen to fall —
does not satisfy that constraint, so the hardware falls back to dense execution and dutifully
multiplies by all your zeros at full price. You get the memory saving and none of the latency
saving. To get speed out of pruning you must either prune in a *pattern the chip can exploit* (2:4,
at some accuracy cost, because you are no longer free to delete the least useful weights) or prune
*structurally* — whole attention heads, whole layers, whole channels — which leaves a smaller dense
model that every runtime already knows how to make fast.

### A worked budget: 70B on one 24 GB card

You have a single RTX 4090 (24 GB) and you want [Llama 3.1 70B](https://howaiworks.ai/models/llama). Its
70,553,706,496 parameters occupy **141 GB** in BF16. The budget gap is 141 / 24 ≈ **5.9x**.

**Quantize it.** INT4 is half a byte per parameter: 70.55B × 0.5 = **35.3 GB**. That is a 4x cut
and it is not enough — you are still 11 GB over, before a single byte of [KV
cache](https://howaiworks.ai/glossary/kv-cache). Sixteen bits to four is essentially the whole useful range of the dial;
below 4-bit, quality falls away quickly.

**Prune it.** NVIDIA's Llama-3_1-Nemotron-51B is Llama 3.1 70B put through neural architecture
search and block-wise distillation, and it is genuinely faster — **2.2x** the inference speed of
the 70B. At INT4 it needs 51B × 0.5 = **25.5 GB**. Still over budget. You have now spent a
distillation run and you still do not fit.

**Change models.** Gemma 3 27B in Google's official INT4 QAT checkpoint is **14.1 GB**, down from
**54 GB** in BF16 — and Gemma 3 is itself a distilled model. It fits with roughly 10 GB left over,
which is not slack: it is KV cache, and KV cache is batch size.

The conclusion is the point. Two of the four methods keep your model, and between them they buy
about 4x. When the gap is bigger than that, more compression is the wrong answer, and the right one
is a method that produces a *different model*. Recognising which side of that line you are on is
most of the decision.

## Types

**[Quantization](https://howaiworks.ai/glossary/quantization) — fewer bits per weight.** The cheapest lever by a wide
margin: GPTQ quantizes a 175B model in about **four GPU hours**, no training data, no gradients. It
keeps every parameter and shrinks each one, and because token generation is bottlenecked on reading
weights out of memory, it makes the model faster as well as smaller. Start here; the quantization
page has the number formats, the accuracy evidence and the outlier problem.

**Pruning — fewer weights.** Two families, and the difference decides whether you get a speedup at
all. *Unstructured* pruning zeroes individual weights and is the most accurate per weight removed;
*structured* pruning removes whole heads, channels or layers and yields a smaller dense model. The
semi-structured 2:4 pattern is the compromise the hardware was built for. Pruning aggressively
usually requires a recovery training run, at which point you are doing distillation anyway.

**[Knowledge distillation](https://howaiworks.ai/glossary/knowledge-distillation) — a smaller model, trained to imitate.**
The expensive lever, and the only one with no ceiling. NVIDIA's Minitron work pruned and distilled a
15B model down to 8B and 4B using **fewer than 3% of the original training tokens** — up to **40x
fewer tokens per model** than training from scratch — and the distilled 4B scored **16% better on
MMLU** than the same architecture trained from scratch. That last figure is the case for
distillation in one number: a small model taught by a big one beats the same small model taught by
the data.

**Low-rank factorization — a big matrix as the product of two small ones.** Replace a weight matrix
`W` of size *m × n* with `A · B`, where `A` is *m × r* and `B` is *r × n*. Llama 3.1 70B's hidden
dimension is 8,192, so a query projection is 8,192 × 8,192 = **67.1M parameters**; the factored
version costs *r* × (8,192 + 8,192) = 16,384*r*. Break-even is at rank 4,096 — *half* the dimension
— and a 4x saving needs rank 1,024, which requires that essentially all of the matrix's information
lives in an eighth of its directions. In transformers it usually does not: LLM weight matrices are
close to full-rank, which is why plain SVD on the weights loses so much. Where low-rank genuinely
works is when it is designed in rather than applied after: DeepSeek-V2's [Multi-head Latent Attention](https://howaiworks.ai/glossary/grouped-query-attention)
compresses keys and values into a low-rank latent vector and cuts the KV cache by **93.3%**, lifting
maximum generation throughput **5.76x**.

That is the complete taxonomy. Anything else you read about — weight sharing, Huffman coding of the
weight file, early exit — is either a refinement of one of these four or is not compression.

## Real-World Applications

**Fitting a frontier-class model on one accelerator.** Llama-3_1-Nemotron-51B exists for one reason:
NVIDIA pruned and distilled Llama 3.1 70B so that the result would fit and run well on a single
H100-80GB, reporting 4x larger workloads on that one GPU. That is not a benchmark result, it is a
procurement decision — one GPU instead of two, for the life of the deployment.

**Local models are a compression artifact.** Google states plainly that the int4 QAT
[Gemma 3](https://howaiworks.ai/models/gemma) 27B "fits comfortably on a single desktop NVIDIA RTX 3090" at 14.1 GB, and
that the 12B fits an 8 GB laptop GPU at 6.6 GB. Without compression there is no consumer tier —
there is a data centre or nothing.

**[Edge AI](https://howaiworks.ai/glossary/edge-ai) has no uncompressed path.** The NPU in a phone or a camera executes
INT8 and INT4; an FP32 model there is not slow, it is unrunnable. Compression is not an optimisation
on that hardware, it is the port.

## Key Concepts

- **Smaller and faster are two different purchases.** A `W4A16` scheme quantizes the weights but
  runs the maths in 16-bit; an unstructured 50% pruned model stores half the weights and runs at
  dense speed. Both halve memory; neither reliably halves latency. Know which one you are buying.
- **The weights are not the whole budget.** At long context the [KV cache](https://howaiworks.ai/glossary/kv-cache) can
  rival or exceed the weights, and none of the four methods touch it — compressing it is a separate
  decision with separate trade-offs.
- **A compressed model is a new model.** It has its own failure modes, and your existing evaluations
  were run on something else.

## Challenges

**You hit the memory target and the model is no faster.** This is the most common and most
demoralising outcome. It happens when the thing you compressed was not the thing that was slow: you
pruned unstructured and the tensor cores ignored it, or you quantized weights on a workload that was
compute-bound rather than [memory-bound](https://howaiworks.ai/glossary/memory-wall) to begin with. Measure first, and be
explicit about whether you are buying capacity or latency.

**The accuracy loss lands exactly where your benchmark cannot see it.** A compressed model can score
within a point or two of the original on aggregate evaluations while behaving measurably differently
on individual cases, with the damage concentrated in the tail — rare scripts, long reasoning chains,
unusual formats. The [quantization](https://howaiworks.ai/glossary/quantization) page documents the evidence for this in
detail and it generalises to the other three methods. The remedy is not another benchmark: it is a
paired comparison of the compressed and original models on your own traffic, counting disagreements.

**Recovery training is the hidden cost of pruning.** One-shot pruning at 50% is genuinely cheap. Get
greedy — 70%, or structural pruning that removes whole layers — and you need a retraining run to
claw the quality back, which means data, compute and a pipeline. Pruning has then quietly become
distillation with extra steps, and you should have priced it that way from the start.

**What breaks if you get this wrong:** you compress to fit a memory budget, ship, and find
throughput unchanged because the sparsity was invisible to the hardware — or that the model is fine
on your eval suite and worse at the one thing your users do. Both are cheap to avoid and expensive
to discover in production.

## Future Trends

Compression is moving upstream, from the community to the lab. Gemma 3's official QAT checkpoints
and NVIDIA's Nemotron models are the template: the organisation that trained the model is the only
one that can prune, distil or quantize it properly, because only it has the data and the pipeline.
The compressed checkpoint is turning from something enthusiasts produced *after* release into a
supported artifact that ships on day one with its own evaluation numbers.

The deeper shift is from compressing a model to designing one that never needed compressing.
Multi-head Latent Attention and [mixture-of-experts](https://howaiworks.ai/glossary/mixture-of-experts) routing are both,
structurally, compression techniques — low-rank and sparsity respectively — moved out of the
post-processing step and into the architecture, where they can be trained rather than retrofitted.
The efficiency you can build in beats the efficiency you can bolt on.

## Frequently Asked Questions

### Which compression method should I use?

It depends on which resource you are short of. If the model does not fit in memory, quantization is the first and cheapest lever. If you need it to run faster, only structured pruning or a genuinely smaller model will do it. If the gap between what you have and what you can afford is more than about 4x, no amount of compressing your existing model will close it and you need a smaller model instead.

### Can you combine quantization, pruning and distillation?

Yes, and the standard pipeline runs in that order backwards: prune first, retrain or distil to recover the quality you lost, and quantize last. Quantization is a final encoding step whose calibration should see the weights you are actually going to ship. The 2015 Deep Compression paper established this composition, reaching 35x on AlexNet with pruning plus quantization plus Huffman coding.

### Why did pruning my model not make it any faster?

Because the zeros are in the wrong places. GPU sparse tensor cores accelerate exactly one pattern — 2:4, meaning two of every four consecutive weights are zero — and give up to 2x on that matmul. An irregular 50% mask does not match the pattern, so the hardware falls back to dense execution and multiplies by your zeros at full price. Unstructured sparsity buys storage, not latency.

### Does model compression hurt accuracy?

Some, and the damage is rarely where the benchmark looks. Aggregate scores can hold within a point or two while long-tail behaviour — rare languages, long reasoning chains, unusual formats — degrades much more. The quantization page documents this in detail. Gate a compressed model on a paired comparison against the original on your own traffic, not on a leaderboard average.

### Is LoRA a model compression technique?

No. LoRA uses a low-rank matrix to make fine-tuning cheap by training a small number of extra parameters; the base model stays exactly as large as it was. Low-rank factorization as compression replaces the original weight matrix with two smaller factors, which is a different operation with a different goal.

## Related

### Related terms

- [Quantization](https://howaiworks.ai/glossary/quantization)
- [Knowledge Distillation](https://howaiworks.ai/glossary/knowledge-distillation)
- [Model Size](https://howaiworks.ai/glossary/model-size)
- [Parameters](https://howaiworks.ai/glossary/parameters)
- [Edge AI](https://howaiworks.ai/glossary/edge-ai)
- [Inference](https://howaiworks.ai/glossary/inference)

---

Source: https://howaiworks.ai/glossary/model-compression — HowAIWorks.ai