A small student learns from a big teacher’s soft answers. Divide the teacher’s logits by a temperature T and take softmax: the label says only “cat”, but the softened distribution also says how much like a dog, how much like a fox. Those relative probabilities of the wrong classes — Hinton’s “dark knowledge” — carry more than the one-hot label alone. Terms go in, the softmax runs, the soft target comes out. Blue builds & witnesses; red attacks & breaks.
source Hinton, Vinyals & Dean, Distilling the Knowledge in a Neural Network (2015) — arxiv.org/abs/1503.02531. Rendered, not quoted.
One line, four provable properties:
pi(T) = exp(li/T) ÷ Σj exp(lj/T)
P1 it is a valid distribution — every pi ≥ 0 and they sum to 1, for every T > 0. P2 as T → ∞ it flattens to uniform. P3 as T → 0+ it sharpens to a one-hot spike on the teacher’s argmax. P4 it is monotone in each logit — raise a logit, its probability rises.
Distillation is the temperature pointed at a teacher’s output. There, T reshapes one model’s own sampling; here, the same knob softens a fixed teacher so a second model can read the wrong-class probabilities and copy the whole shape, not just the peak.
The soft probabilities of the wrong answers are the signal — more than the label alone. That single move — a distribution is worth more than its argmax — is the premise the next sphere builds on.
The blue team’s live check: recompute the sealed teacher’s soft target and confirm it still carries information (entropy > 0) and still preserves the class order of the logits. If red collapses T, this badge is where it shows.
A trained teacher scores five classes with raw logits (pre-softmax). This one is confident about C0 — but it is not blind to the rest:
| class | logit li | reads as |
|---|---|---|
| C0 ★ | 3.0 | the answer (argmax) |
| C1 | 1.0 | plausible |
| C4 | 0.8 | plausible |
| C2 | 0.3 | less so |
| C3 | -0.5 | unlikely |
The one-hot label keeps only the first row. The logits keep the ordering of the rest — C1 > C4 > C2 > C3 — and that ordering is what you feed the panel below.
Raise T and the answer softens — the wrong-class probabilities rise and the dark knowledge gets louder. T→1 recovers the raw teacher; T→0 would collapse it back to the hard label.
Move the slider — every probability and the entropy are computed from the softmax on the spot, never looked up.
What the machine produces, proven: a probability distribution that (a) always sums to 1, (b) at the distillation temperature T=4 has entropy strictly above 0 — strictly more than the hard label’s zero — and (c) ranks the wrong classes in the teacher’s own logit order. That extra structure is the “dark knowledge” the student trains against; at test time T returns to 1.
The blue witness (left) confirms this live; the red team (right) tries to erase it.
T is a hyperparameter, not a law — too low and it degenerates to hard labels, too high and it washes out into uniform noise. Distillation also needs the teacher’s full logits on a transfer set; in a pure top-1 API setting there is no dark knowledge to read. It compresses nothing on its own — the student must have the capacity to fit the softened function.
“The student just copies the teacher’s prediction.” Cut. It fits the teacher’s whole distribution — the relative probabilities of the wrong classes — not the top-1 label.
“Temperature is only used at test time.” Kept, corrected. T scales the logits of both teacher and student during transfer; at inference T returns to 1.
“Soft targets always beat hard labels.” Cut. Only when the teacher is well-calibrated and real dark knowledge exists; otherwise you distill its mistakes.
The red team’s move: drive the teacher’s target temperature to T → 0, collapsing the soft target to a hard one-hot. The dark knowledge is destroyed — its entropy falls to 0. The blue witness (window 7) is watching.
Collapse the teacher target and it becomes indistinguishable from the plain label — the witness recomputes its entropy, finds 0, and turns red. Nothing is faked; the attack is real and it is caught.