The distance that is not a distance. Feed it two beliefs over the same alphabet — P, the world; Q, your model of it — and it returns the extra cost, in nats, of coding reality with the wrong book. It is never negative, it is zero only when Q = P, and it is not symmetric: swapping the two gives a different number. Cross-entropy, RLHF and the VAE all minimize it. Down the center, data flows: the two distributions go in, the engine sums p·ln(p/q), the total comes out. The blue team builds and defends it; the red team tries to break it.
source Kullback & Leibler, On Information and Sufficiency, Ann. Math. Statist. 22(1), 1951, 79–86 — open access at projecteuclid.org · doi:10.1214/aoms/1177729694. AMBER no confirmed archive.org/details id found; Project Euclid is the primary open-access scan. Rendered, not quoted.
Over a shared alphabet, the divergence is a single sum:
D(P∥Q) = ∑x p(x)·ln( p(x) / q(x) )
with the convention 0·ln0 = 0. Three properties fall straight out of it and the engine checks all three live:
L1 Gibbs — D ≥ 0 always (convexity of −ln). L2 identity — D(P∥P) = 0 exactly, and only then. L3 asymmetry — D(P∥Q) ≠ D(Q∥P) in general, so it is a divergence, not a metric.
Current pair, its Gibbs floor and the live asymmetry gap:
Shannon's entropy H(P) is the cost of the best code for P. Cross-entropy H(P,Q) is the cost of coding P with Q's code. Their difference is exactly this sphere:
D(P∥Q) = H(P,Q) − H(P)
KL is entropy given a direction — the gap between the code you used (Q) and the code you should have used (P). Because H(P) does not depend on Q, minimizing cross-entropy over your model is minimizing KL. That is the objective under nearly every trained model. Each sphere is the next one's premise.
The blue team's live check: re-sweep 400 fixed-seed pairs, confirm the Gibbs floor holds (min D ≥ 0) and the decomposition is exact. If red flips the log, this badge is where it shows.
The input is two probability distributions over the same alphabet {a,b,c,d}. P is the truth (the reference); Q is your model or code. Each column is auto-normalized to sum to 1 — type any non-negative weights.
KL requires absolute continuity: wherever P>0, Q must be >0, or the cost is infinite. That is the first thing the engine tells you.
Per-symbol contribution p·ln(p/q) — positive where Q under-weights a likely symbol, negative where it over-weights it. The total is the divergence, in nats.
| x | p | q | p·ln(p/q) |
|---|
Change any weight — the sum is recomputed from ln(p/q) on the spot, never looked up.
What the machine produces, proven: the total D(P∥Q) in nats above, and the three invariants that make it meaningful — ≥ 0 over 400 seeded pairs, = 0 exactly when Q = P (to 1e−12), and asymmetric, with the exact decomposition H(P,Q)−H(P).
The blue team's witness (left) confirms these live; the red team (right) tries to make the floor negative.
Worse, the direction is a modeling choice with real consequences: forward KL D(P∥Q) is mode-covering (Q must smear over all of P's support); reverse KL D(Q∥P) is mode-seeking (Q can collapse onto one peak). Same data, opposite fits. KL is not the gap between beliefs; it is one of them.
"KL divergence is the distance between two distributions." Cut. A distance is symmetric and obeys the triangle inequality; KL does neither. It is a divergence — both facts are computed in the machine.
"KL(P∥Q) = KL(Q∥P)." Cut. The engine shows both directions side by side; for the default pair they differ. Averaging the two gives the (symmetric) Jensen–Shannon divergence — a different object.
"Cross-entropy loss is a different objective from KL." Kept, corrected. They share an argmin: H(P,Q) = H(P) + D(P∥Q), and H(P) is constant in Q, so minimizing cross-entropy is minimizing KL.
The red team's move: flip the log argument to ln(q/p), so the engine computes ∑ p·ln(q/p) = −D. By Gibbs that is ≤ 0 — yet D(P∥P) still reads 0, so the identity check is fooled. Only the witness (window 7), which watches the floor, catches it.
Flip it and the Gibbs floor goes negative while the diagonal stays 0. The witness recomputes, sees min D < 0, and turns red. Nothing is faked; the attack is real and it is caught.