Model Compression

Definition

Model compression is making an already-trained model cheaper to run, and choosing a method is really choosing which resource you are short of. There are four, and they do not do the same job: quantization cuts the bytes per weight, pruning cuts the number of weights, knowledge distillation trains a different, smaller model to imitate the one you have, and low-rank factorization replaces one big matrix with the product of two small ones.

The distinction that matters before any of the technical detail is this: quantization and pruning keep your model, and distillation and low-rank factorization hand you a new one. A new model means new weights to ship, a training run to pay for, and an evaluation suite that no longer applies. Most teams reach for the first pair because the second pair is expensive — and then discover that the first pair cannot close the gap they have.

How It Works

Every method removes something different, and that is the whole basis of the choice. Quantization leaves all 70 billion parameters in place and stores each in 4 bits instead of 16. Pruning leaves the format alone and deletes parameters. Distillation deletes neither: it trains a smaller network from scratch, supervised by the big one's outputs. Low-rank factorization keeps the layer but re-expresses its weight matrix in fewer numbers.

They compose, and the order is not arbitrary: prune, then retrain or distil to recover what you broke, then quantize last. Quantization is a final lossy encoding, and its calibration should see the weights you are actually shipping — quantize first and you will re-damage the result when you prune. This is old news, not a new idea: the 2015 Deep Compression paper ran exactly this pipeline (pruning, then trained quantization, then Huffman coding) and shrank AlexNet 35x, from 240 MB to 6.9 MB, with no loss of accuracy, and VGG-16 49x.

The trap: sparsity your hardware cannot use

SparseGPT can prune OPT-175B to 50% sparsity in one shot with no retraining and negligible perplexity change — half the weights, gone, for free. It sounds like a two-times speedup. It is usually a zero-times speedup, and this is the single most useful thing to know before you prune.

GPU sparse tensor cores accelerate exactly one pattern: 2:4, meaning two of every four consecutive weights must be zero, which is worth up to 2x throughput on that matrix multiply. An unstructured mask — the naturally best 50% of weights to delete, wherever they happen to fall — does not satisfy that constraint, so the hardware falls back to dense execution and dutifully multiplies by all your zeros at full price. You get the memory saving and none of the latency saving. To get speed out of pruning you must either prune in a pattern the chip can exploit (2:4, at some accuracy cost, because you are no longer free to delete the least useful weights) or prune structurally — whole attention heads, whole layers, whole channels — which leaves a smaller dense model that every runtime already knows how to make fast.

A worked budget: 70B on one 24 GB card

You have a single RTX 4090 (24 GB) and you want Llama 3.1 70B. Its 70,553,706,496 parameters occupy 141 GB in BF16. The budget gap is 141 / 24 ≈ 5.9x.

Quantize it. INT4 is half a byte per parameter: 70.55B × 0.5 = 35.3 GB. That is a 4x cut and it is not enough — you are still 11 GB over, before a single byte of KV cache. Sixteen bits to four is essentially the whole useful range of the dial; below 4-bit, quality falls away quickly.

Prune it. NVIDIA's Llama-3_1-Nemotron-51B is Llama 3.1 70B put through neural architecture search and block-wise distillation, and it is genuinely faster — 2.2x the inference speed of the 70B. At INT4 it needs 51B × 0.5 = 25.5 GB. Still over budget. You have now spent a distillation run and you still do not fit.

Change models. Gemma 3 27B in Google's official INT4 QAT checkpoint is 14.1 GB, down from 54 GB in BF16 — and Gemma 3 is itself a distilled model. It fits with roughly 10 GB left over, which is not slack: it is KV cache, and KV cache is batch size.

The conclusion is the point. Two of the four methods keep your model, and between them they buy about 4x. When the gap is bigger than that, more compression is the wrong answer, and the right one is a method that produces a different model. Recognising which side of that line you are on is most of the decision.

Types

Quantization — fewer bits per weight. The cheapest lever by a wide margin: GPTQ quantizes a 175B model in about four GPU hours, no training data, no gradients. It keeps every parameter and shrinks each one, and because token generation is bottlenecked on reading weights out of memory, it makes the model faster as well as smaller. Start here; the quantization page has the number formats, the accuracy evidence and the outlier problem.

Pruning — fewer weights. Two families, and the difference decides whether you get a speedup at all. Unstructured pruning zeroes individual weights and is the most accurate per weight removed; structured pruning removes whole heads, channels or layers and yields a smaller dense model. The semi-structured 2:4 pattern is the compromise the hardware was built for. Pruning aggressively usually requires a recovery training run, at which point you are doing distillation anyway.

Knowledge distillation — a smaller model, trained to imitate. The expensive lever, and the only one with no ceiling. NVIDIA's Minitron work pruned and distilled a 15B model down to 8B and 4B using fewer than 3% of the original training tokens — up to 40x fewer tokens per model than training from scratch — and the distilled 4B scored 16% better on MMLU than the same architecture trained from scratch. That last figure is the case for distillation in one number: a small model taught by a big one beats the same small model taught by the data.

Low-rank factorization — a big matrix as the product of two small ones. Replace a weight matrix W of size m × n with A · B, where A is m × r and B is r × n. Llama 3.1 70B's hidden dimension is 8,192, so a query projection is 8,192 × 8,192 = 67.1M parameters; the factored version costs r × (8,192 + 8,192) = 16,384r. Break-even is at rank 4,096 — half the dimension — and a 4x saving needs rank 1,024, which requires that essentially all of the matrix's information lives in an eighth of its directions. In transformers it usually does not: LLM weight matrices are close to full-rank, which is why plain SVD on the weights loses so much. Where low-rank genuinely works is when it is designed in rather than applied after: DeepSeek-V2's Multi-head Latent Attention compresses keys and values into a low-rank latent vector and cuts the KV cache by 93.3%, lifting maximum generation throughput 5.76x.

That is the complete taxonomy. Anything else you read about — weight sharing, Huffman coding of the weight file, early exit — is either a refinement of one of these four or is not compression.

Real-World Applications

Fitting a frontier-class model on one accelerator. Llama-3_1-Nemotron-51B exists for one reason: NVIDIA pruned and distilled Llama 3.1 70B so that the result would fit and run well on a single H100-80GB, reporting 4x larger workloads on that one GPU. That is not a benchmark result, it is a procurement decision — one GPU instead of two, for the life of the deployment.

Local models are a compression artifact. Google states plainly that the int4 QAT Gemma 3 27B "fits comfortably on a single desktop NVIDIA RTX 3090" at 14.1 GB, and that the 12B fits an 8 GB laptop GPU at 6.6 GB. Without compression there is no consumer tier — there is a data centre or nothing.

Edge AI has no uncompressed path. The NPU in a phone or a camera executes INT8 and INT4; an FP32 model there is not slow, it is unrunnable. Compression is not an optimisation on that hardware, it is the port.

Key Concepts

• Smaller and faster are two different purchases. A W4A16 scheme quantizes the weights but runs the maths in 16-bit; an unstructured 50% pruned model stores half the weights and runs at dense speed. Both halve memory; neither reliably halves latency. Know which one you are buying.
• The weights are not the whole budget. At long context the KV cache can rival or exceed the weights, and none of the four methods touch it — compressing it is a separate decision with separate trade-offs.
• A compressed model is a new model. It has its own failure modes, and your existing evaluations were run on something else.

Challenges

You hit the memory target and the model is no faster. This is the most common and most demoralising outcome. It happens when the thing you compressed was not the thing that was slow: you pruned unstructured and the tensor cores ignored it, or you quantized weights on a workload that was compute-bound rather than memory-bound to begin with. Measure first, and be explicit about whether you are buying capacity or latency.

The accuracy loss lands exactly where your benchmark cannot see it. A compressed model can score within a point or two of the original on aggregate evaluations while behaving measurably differently on individual cases, with the damage concentrated in the tail — rare scripts, long reasoning chains, unusual formats. The quantization page documents the evidence for this in detail and it generalises to the other three methods. The remedy is not another benchmark: it is a paired comparison of the compressed and original models on your own traffic, counting disagreements.

Recovery training is the hidden cost of pruning. One-shot pruning at 50% is genuinely cheap. Get greedy — 70%, or structural pruning that removes whole layers — and you need a retraining run to claw the quality back, which means data, compute and a pipeline. Pruning has then quietly become distillation with extra steps, and you should have priced it that way from the start.

What breaks if you get this wrong: you compress to fit a memory budget, ship, and find throughput unchanged because the sparsity was invisible to the hardware — or that the model is fine on your eval suite and worse at the one thing your users do. Both are cheap to avoid and expensive to discover in production.

Future Trends

Compression is moving upstream, from the community to the lab. Gemma 3's official QAT checkpoints and NVIDIA's Nemotron models are the template: the organisation that trained the model is the only one that can prune, distil or quantize it properly, because only it has the data and the pipeline. The compressed checkpoint is turning from something enthusiasts produced after release into a supported artifact that ships on day one with its own evaluation numbers.

The deeper shift is from compressing a model to designing one that never needed compressing. Multi-head Latent Attention and mixture-of-experts routing are both, structurally, compression techniques — low-rank and sparsity respectively — moved out of the post-processing step and into the architecture, where they can be trained rather than retrofitted. The efficiency you can build in beats the efficiency you can bolt on.