Knowledge Distillation

Definition

Knowledge distillation trains a small model — the student — on the entire probability distribution a large model — the teacher — produces for each example, instead of on the single correct answer in the training data. That is the whole idea, and the reason it works is the part most explanations skip. A label says an image is a 7 and says nothing else. A trained teacher says it is a 7 with probability 0.9925, a 1 with 0.0041, a 9 with 0.0016, and a 3 with 0.000045. Those tiny numbers on the wrong classes are not noise: they say this particular 7 has a little of a 1 about it and rather less of a 9, which is a fact about how digits resemble each other that no label in the dataset contains. Geoffrey Hinton named it the dark knowledge, and it is exactly what the student gets from the teacher that the data could not give it.

The consequence is the counter-intuitive result at the centre of the technique: a small model taught by a big model can beat the same small model taught by the data. Hinton, Vinyals and Dean (2015) showed it on MNIST in three measurements. A large, heavily regularised net with two hidden layers of 1,200 units made 67 test errors. A smaller net with two hidden layers of 800 units and no regularisation made 146. The same 800-unit net, regularised by nothing except the extra task of matching the large net's softened outputs, made 74 — recovering nearly all of a gap that its own architecture supposedly could not close.

Distillation is one of the four things people mean by model compression, and the only one that hands you a genuinely different model rather than a re-encoded or perforated copy of the one you had; that page owns the choice between the four, this one owns the mechanism. It needs separating from the neighbour on the other side too. Distillation is almost always implemented as a fine-tuning run, but fine-tuning describes which weights you update while distillation describes where the targets come from — and confusing them is how a team does the expensive half, generating a million outputs from a frontier model, while discarding the half that makes it work.

How It Works

Softening the teacher

The teacher's raw outputs are logits, unnormalised scores turned into probabilities by the softmax. Distillation applies the softmax with a temperature T in the denominator of every exponent:

p_i = exp(z_i / T) / sum_j exp(z_j / T)

At T = 1 this is the ordinary softmax. Raising T flattens the distribution toward uniform; lowering it sharpens toward one-hot. Here is the same teacher's opinion about one image, at four temperatures — computed by the code at the end of this page, from the logit vector [-1.2, 2.5, -0.5, -2.0, 0.3, -1.5, -0.8, 8.0, 0.9, 1.6]:

The first two rows are the argument. At T = 1 the teacher's distribution is almost exactly the hard label: 99.25% of the mass sits on the answer, entropy is 0.055 nats against a maximum of 2.303, and the whole structure among the nine wrong classes is squeezed into three-quarters of one percent. The ordering is there — 1 beats 9 beats 8 — but it contributes so little to a cross-entropy loss that gradient descent never notices. At T = 5 that same ordering carries 65% of the mass and dominates the loss. Nothing was added; softening only made visible what the teacher already believed. Push too far and it goes the other way: at T = 20 the entropy is 2.291 nats, within 0.5% of a uniform distribution over ten classes, and the teacher has been softened into saying nothing at all.

This is the formula the temperature page describes, running backwards. There, temperature is applied at inference and makes the model's sampled output more random. Here nothing is sampled: T is applied to a fixed teacher distribution to turn it into a richer training target, and set back to 1 the moment the student ships. Same equation, opposite purpose.

Why the loss is multiplied by T²

This is the detail that separates people who have implemented distillation from people who have read about it. The soft-target loss is cross-entropy between the student's softmax at temperature T and the teacher's softmax at the same T. Differentiate it with respect to a student logit and you get (1/T)(q_i − p_i) — one factor of 1/T straight from the chain rule. The second factor hides inside (q_i − p_i): raising T flattens both distributions toward uniform, so the difference between them shrinks like 1/T as well. Multiply them and the gradient dies as 1/T².

Hinton, Vinyals and Dean state the correction in a single sentence: "Since the magnitudes of the gradients produced by the soft targets scale as 1/T² it is important to multiply them by T² when using both hard and soft targets." So the objective a real implementation minimises is

L = alpha * T^2 * CE(student at T, teacher at T) + (1 - alpha) * CE(student at 1, hard label)

with the hard-label term always computed at T = 1. Leave the T² out and the temperature stops being a temperature: raising it from 4 to 8 quietly divides the soft term's contribution by roughly four, so the sweep you thought was tuning softness was mostly tuning alpha. The second code block below measures the decay directly — the gradient norm falls by 4.22× for a doubling of T from 32 to 64, converging on the factor of exactly four that 1/T² predicts.

The experiment that shows dark knowledge is real

The MNIST numbers above show a student catching up. The follow-up experiment in the same paper shows why, and it is the most quoted and most garbled result in the field. Hinton and colleagues removed every example of the digit 3 from the transfer set, so the student never once saw a 3 during distillation. It still made only 206 test errors, of which 133 were on the 1,010 threes in the test set. The student had learned what a 3 looks like entirely from the small probabilities the teacher assigned to the 3 class while looking at every other digit.

The part popular summaries drop: those 206 errors are before a correction. Most of the remaining damage was one miscalibrated output bias, because no example had ever wanted 3 as the answer. Raise that single bias by 3.5 and the student makes 109 errors of which 14 are on 3s — 98.6% of test 3s correct, having never seen a 3. Retellings that omit the bias adjustment turn a reproducible result into a magical one. Pushed harder, with a transfer set of only 7s and 8s, the effect degrades rather than vanishing: 47.3% test error, falling to 13.2% once the biases for 7 and 8 are lowered by 7.6.

One practical consequence follows immediately and is worth more than it first appears: the transfer set does not need labels. The teacher supplies the targets, so any unlabelled data from roughly the right distribution will do. Distillation turns a labelling problem into an inference problem.

What it costs, and when it pays back

A forward pass costs roughly 2P FLOPs per token for a model with P parameters; a training step costs roughly 6P. Distilling a 7B student from a 70B teacher over a transfer set of D tokens therefore costs 2 × 70e9 × D to run the teacher across the set, plus 6 × 7e9 × D to train the student. At D = 1 trillion tokens that is 1.4e23 + 4.2e22 = 1.82e23 FLOPs, paid once. Every token you afterwards serve from the student instead of the teacher saves 2 × (70 − 7) × 10⁹ = 1.26e11 FLOPs. Divide: break-even at 1.44 trillion served tokens.

The ratio is the durable part, because the absolute numbers cancel. Break-even measured in units of the transfer set is (3·P_student + P_teacher) / (P_teacher − P_student), which is 13/9 = 1.44 for a 10× compression and 103/99 = 1.04 for a 100× one. You pay the distillation bill back after serving roughly as many tokens as you distilled on, and the harder you compress the closer that multiple gets to 1. This is why distillation is a serving decision rather than a research one: nearly free for anything you will run at volume, never worth it for anything you will run once.

Types

Distillation is sorted along two independent axes, and the standard survey taxonomy (Gou et al., 2021) covers the first: what is matched.

Response-based distillation matches the output distribution — everything described above. Its decisive property is that teacher and student share nothing but a label set or vocabulary, so a 70B transformer can teach a 1B one and an ensemble can teach a single net. This is what essentially all production distillation does, and its simplicity is the reason it survived.

Feature-based distillation matches intermediate activations, pushing the student to build similar internal representations rather than only similar answers (FitNets, Romero et al., 2015). It carries more signal per example and charges two things response-based distillation never does: a learned projection whenever hidden sizes differ, and a decision about which student layer imitates which teacher layer — a choice with no principled answer and a real effect on the result. It earns that complexity when the student is deep and thin, where matching only the final answer leaves the middle of the network unsupervised.

Relation-based distillation matches relationships rather than values: pairwise distances between examples in the teacher's representation space, or correlations across layers. The appeal is dimensional independence — a distance between two examples survives a change of hidden size that an activation vector does not.

The second axis is when. Offline distillation uses a frozen, already-trained teacher and is the default; online distillation trains teacher and student together, removing the separate teacher run at the cost of a weaker teacher early on. Self-distillation uses a model as its own teacher, and it is the case that proves the whole thing is about the target rather than the capacity: Furlanello et al. (2018) trained students with identical architecture to their teachers and reported that these Born-Again Networks "outperform their teachers significantly," reaching 3.5% validation error on CIFAR-10 and 15.5% on CIFAR-100 with DenseNets. No compression happened at all; the only thing that changed was what the model was trained against.

Real-World Applications

DistilBERT is still the cleanest published demonstration. Sanh et al. (2019) halved BERT-base's layer count from 12 to 6, initialised the student by taking one teacher layer out of every two, and trained with a triple loss combining distillation, masked language modelling and a cosine-distance term on hidden states. The abstract's claim — "reduce the size of a BERT model by 40%, while retaining 97% of its language understanding capabilities and being 60% faster" — is exactly right and routinely mis-cited. The 40% is parameters: 66M against 110M. The 97% is a macro-average over the GLUE dev sets: 77.0 against 79.5. And the 60% is one specific measurement — a full pass over the STS-B development set on a CPU (Intel Xeon E5-2690 v3) at batch size 1, in 410 seconds against 668 — not a GPU throughput figure and not a guarantee about your workload. The paper separately measured a question-answering app on an iPhone 7 Plus at 71% faster with a 207 MB model, which is the number to quote if you actually care about a phone.

Gemma 3 shows the technique moved from post-processing into pre-training. Google's technical report (March 2025) makes distillation the pre-training objective for every size in the family, across 2T tokens for the 1B model up to 14T for the 27B: "We sample 256 logits per token, weighted by teacher probabilities. The student learns the teacher's distribution within these samples via cross-entropy loss." Against a vocabulary of 262k entries, 256 sampled logits is under 0.1% of the distribution — and that thin slice, applied to trillions of tokens, produces the checkpoints the Gemma family ships. It is the dark-knowledge argument at industrial scale, with the engineering compromise made explicit.

DeepSeek-R1's distilled series is the strongest recent evidence that the teacher's outputs beat the alternatives. In the R1 technical report (January 2025), DeepSeek took Qwen2.5-32B and trained it two ways. Large-scale reinforcement learning on maths, code and STEM data for over 10,000 steps produced 47.0% pass@1 on AIME 2024. Plain supervised fine-tuning on 800k samples generated by R1 — no RL at all — produced 72.6%, along with 94.3% on MATH-500 against 91.6%. Same base model, same evaluation, and the imitation run won by 25 points. The authors' conclusion is unusually blunt: "distilling more powerful models into smaller ones yields excellent results, whereas smaller models relying on the large-scale RL mentioned in this paper require enormous computational power and may not even achieve the performance of distillation."

On-device deployment is where most of this ends up. The standard pipeline for edge AI is train big, distil into a student the device can hold, then quantize — and distillation is the step that changes the model's shape rather than its encoding, so it is the one that decides whether the target hardware is reachable at all.

Key Concepts

• Dark knowledge: the relative probabilities the teacher assigns to wrong answers. It is the entire payload, it is invisible at T = 1, and it is destroyed by taking an argmax.
• Transfer set: the data the teacher is run over to produce targets. It needs no labels, need not be the teacher's training data, and its coverage — not its size — is what limits the student.
• Two hyperparameters that interact: alpha weights soft targets against hard labels, T sets how much structure is visible — and the second silently changes the first unless you apply the T² correction.
• Capacity gap: the distance between teacher and student size. Wider is not better, and past a point it actively hurts.
• Sequence-level distillation: fitting a teacher's sampled text rather than its probabilities. This is what the word now usually means for large language models, and it is a far lossier channel that compensates with volume.

Challenges

A bigger teacher is not a better teacher, and the literature does not agree on the fix. Cho and Hariharan (2019) found that "larger models do not often make better teachers," diagnosed it as mismatched capacity — a small student cannot represent the function a very large teacher computes — and proposed stopping the teacher's training early so its distribution stays within reach. Mirzadeh et al. (AAAI 2020) proposed the opposite remedy: insert an intermediate "teacher assistant" and distil in two hops. Cho and Hariharan tested exactly that and reported "typical ways of circumventing this (such as performing a sequence of knowledge distillation steps) to be ineffective." So the capacity gap is a real effect with a contested fix: if your student is 50× smaller than your teacher, an intermediate model is worth an experiment rather than an assumption.

Everything the teacher believes is transferred, including the wrong parts. Dark knowledge is the teacher's generalisation, so the student inherits its biases, its systematic errors and — for language models — its hallucinated facts, learned as confidently as the correct ones. The ground-truth term is the only thing in the loss pulling the other way, which is a concrete reason to keep alpha below 1 even when the soft targets are obviously more informative.

Modern LLM distillation throws away most of the mechanism above. DeepSeek's 800k samples are text, not distributions: sampling collapses a 100,000-way probability vector into one token, destroying exactly the dark knowledge that makes the technique special. It works anyway, because a long reasoning trace contains thousands of such decisions and the structure re-emerges across the sequence instead of within each step — but that is why sequence-level distillation needs hundreds of thousands of examples where soft-target distillation needed a transfer set. Knowing which one you are doing tells you whether more examples or better targets is the lever.

Logit access is a hard boundary, legal as well as technical. A closed API returns sampled text and at best a handful of top log-probabilities — never the full distribution, never the intermediate activations feature-based methods need. Beyond that, most commercial providers' terms of service prohibit using their outputs to train a competing model. Real soft-target distillation belongs to whoever holds the teacher's weights.

What breaks if you get this wrong: you generate a large corpus from a frontier model, fine-tune a small one on the text, and get a student that matches your generated corpus and nothing beyond it — because sampling discarded the generalisation signal and your transfer set never covered what your users actually send. It looks like a data problem and is really a targets problem.

Future Trends

The clearest shift is the one Gemma 3 makes explicit: distillation is moving from a post-training compression step into the pre-training objective itself. When the small models in a family are distilled from the large one during their trillions of tokens rather than fitted to it afterwards, "the 4B" stops being a compressed 27B and becomes a model that was never trained on hard labels at all. Expect model families to look increasingly like one large teacher with a distilled ladder beneath it.

The second is that reasoning-trace distillation has become the cheapest way to move a capability between models, and its ceiling is now the interesting question. DeepSeek's own second conclusion is the sober one: distillation is "both economical and effective," but "advancing beyond the boundaries of intelligence may still require more powerful base models and larger-scale reinforcement learning." Imitation propagates a capability efficiently; it has never yet created one.

The third is structural. Full-distribution distillation needs logits, closed models do not emit them, and so the gap between what an open-weights teacher and a closed API can teach is widening — which makes open-weights releases disproportionately valuable as teachers, and makes the terms-of-service question a commercial battleground rather than a footnote.

Code Example

The claim at the top of this page is that a teacher's distribution at T = 1 carries almost nothing beyond the label, and that raising the temperature makes the structure among the wrong answers visible. That is checkable in twenty lines of standard library Python — no dependencies, no random draws:

import math

# One teacher's logits for a single MNIST-style image whose true label is 7.
LOGITS = [-1.2, 2.5, -0.5, -2.0, 0.3, -1.5, -0.8, 8.0, 0.9, 1.6]

def softmax(z, T=1.0):
    m = max(x / T for x in z)
    e = [math.exp(x / T - m) for x in z]
    s = sum(e)
    return [v / s for v in e]

print("            " + "".join(f"{d:>8}" for d in range(10)) + "   entropy   mass off top")
hard = [0.0] * 10
hard[7] = 1.0
print("hard label  " + "".join(f"{v:8.4f}" for v in hard) + "     0.000        0.0000")
for T in (1, 3, 5, 20):
    p = softmax(LOGITS, T)
    H = -sum(x * math.log(x) for x in p)
    print(f"T = {T:<8}" + "".join(f"{v:8.4f}" for v in p) + f"     {H:.3f}        {1 - max(p):.4f}")

Real output:

                   0       1       2       3       4       5       6       7       8       9   entropy   mass off top
hard label    0.0000  0.0000  0.0000  0.0000  0.0000  0.0000  0.0000  1.0000  0.0000  0.0000     0.000        0.0000
T = 1         0.0001  0.0041  0.0002  0.0000  0.0004  0.0001  0.0001  0.9925  0.0008  0.0016     0.055        0.0075
T = 3         0.0276  0.0949  0.0349  0.0212  0.0456  0.0250  0.0316  0.5934  0.0557  0.0703     1.520        0.4066
T = 5         0.0554  0.1162  0.0638  0.0472  0.0748  0.0522  0.0600  0.3490  0.0844  0.0970     2.049        0.6510
T = 20        0.0899  0.1081  0.0931  0.0863  0.0969  0.0885  0.0917  0.1424  0.0998  0.1034     2.291        0.8576

The T = 1 row and the hard-label row agree to two decimal places on nine of ten classes. That is the case for softening in one picture: distilling at T = 1 is barely distinguishable from training on the labels you already had.

Now the T² correction, which is the same computation run against a student's logits. The gradient of the soft cross-entropy with respect to student logit i is (q_i − p_i) / T:

import math

TEACHER = [-1.2, 2.5, -0.5, -2.0, 0.3, -1.5, -0.8, 8.0, 0.9, 1.6]
STUDENT = [-0.9, 1.8, -0.3, -1.6, 0.1, -1.1, -0.6, 6.4, 0.7, 1.1]

def softmax(z, T=1.0):
    m = max(x / T for x in z)
    e = [math.exp(x / T - m) for x in z]
    s = sum(e)
    return [v / s for v in e]

# d(soft cross-entropy) / d(student logit i)  =  (1/T) * (q_i - p_i)
prev = None
for T in (2, 4, 8, 16, 32, 64):
    p, q = softmax(TEACHER, T), softmax(STUDENT, T)
    g = math.sqrt(sum(((qi - pi) / T) ** 2 for qi, pi in zip(q, p)))
    drop = f"{prev / g:5.2f}x" if prev else "    -"
    print(f"T = {T:<3} |grad| = {g:.3e}   vs previous row: {drop}   |grad| x T^2 = {g * T * T:.4f}")
    prev = g

Real output:

T = 2   |grad| = 6.181e-02   vs previous row:     -   |grad| x T^2 = 0.2472
T = 4   |grad| = 2.325e-02   vs previous row:  2.66x   |grad| x T^2 = 0.3721
T = 8   |grad| = 4.486e-03   vs previous row:  5.18x   |grad| x T^2 = 0.2871
T = 16  |grad| = 9.143e-04   vs previous row:  4.91x   |grad| x T^2 = 0.2341
T = 32  |grad| = 2.053e-04   vs previous row:  4.45x   |grad| x T^2 = 0.2102
T = 64  |grad| = 4.867e-05   vs previous row:  4.22x   |grad| x T^2 = 0.1993

The middle column is the point. Each doubling of T divides the gradient by close to four, converging on exactly four as the approximation that produces the 1/T² law becomes accurate — while the right-hand column, the same gradient with the correction applied, settles down instead of collapsing. Without that T², a run at T = 16 gives the soft targets roughly 1/64th the pull they had at T = 2, and any conclusion you draw about "the best temperature" is really a conclusion about alpha.

Academic Sources

• "Distilling the Knowledge in a Neural Network" — Hinton, Vinyals and Dean (2015) — soft targets, the T² correction, and the MNIST net that recognises a digit absent from its transfer set.
• "DistilBERT, a distilled version of BERT" — Sanh et al. (2019) — 66M against 110M parameters, 77.0 against 79.5 on GLUE, and the triple loss.
• "Born-Again Neural Networks" — Furlanello et al. (2018) — students identical to their teachers outperform them; distillation is about the target, not the size.
• "FitNets: Hints for Thin Deep Nets" — Romero et al. (2015) — the origin of feature-based distillation.
• "Knowledge Distillation: A Survey" — Gou et al. (2021) — the response / feature / relation taxonomy used above.
• "On the Efficacy of Knowledge Distillation" — Cho and Hariharan (2019) — larger teachers do not make better students; stop the teacher's training early.
• "Improved Knowledge Distillation via Teacher Assistant" — Mirzadeh et al. (AAAI 2020) — the competing remedy for the capacity gap.
• "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning" — DeepSeek-AI (2025) — 800k distilled samples beat 10,000 steps of RL on the same 32B base.
• "Gemma 3 Technical Report" — Gemma Team (2025) — distillation as the pre-training objective, with 256 sampled logits per token.