THE KL DIVERGENCE — a dart

THE KL DIVERGENCE how far one distribution is from another

The measure of how much a distribution q differs from a reference p: D(p∥q) = Σ p·ln(p/q) — the extra bits you pay coding samples from p with a code built for q. It is never negative, zero only when p=q, and crucially asymmetric: D(p∥q) ≠ D(q∥p). It sits under variational inference, VAEs, and the distillation of one model into another.

THE TECHNIQUE D(p∥q) = Σ p·ln(p/q)

The demo shows D(p∥p)=0, D(p∥q)>0, and D(p∥q)≠D(q∥p) — asymmetric (i13 ln via series): live demo

HISTORY & CREDIT Kullback & Leibler · 1951

“KL divergence is a distance.” — it is not: it is asymmetric and breaks the triangle inequality, so it is a divergence, not a metric. cited

the extra bits · the coding cost of using q’s code for p’s data.
the properties · ≥ 0, zero iff p=q, and D(p∥q) ≠ D(q∥p) — asymmetric.
1951 · Kullback & Leibler — relative entropy.

A gap between distributions, with a direction. information

RECOMMEND FOR I-13 asymmetry, on the compiler

On i-13, with p=[.5,.5], q=[.9,.1]: D(p∥p)=0, D(p∥q)=0.51, D(q∥p)=0.37 — positive and asymmetric:

$ i13 run n2_kl-divergence.i13 RUN OK · 2138 step(s) · call depth 23 kl_same = 0 kl_pq = 0.511 kl_qp = 0.368 nonneg = 1 zero_when_equal = 1 asymmetric = 1

Recommend as a NULL — an information measure (B39). KL is a pinned functional of p and q; every correct computation agrees. NULL — how far one distribution is from another.