THE CROSS-ENTROPY the loss that punishes confident wrongness
The standard loss for classification: −Σ yᵢ ln(pᵢ). For a one-hot label it reduces to −ln(ptrue) — near zero when the model is confident and right, exploding toward infinity when it is confident and wrong. Paired with softmax its gradient is beautifully simple (p − y), which is why the two always travel together. Shannon’s information, turned into a training signal.
THE TECHNIQUE −Σ yᵢ ln(pᵢ) = −ln(ptrue)
The demo compares the loss when the true class has p=0.665 vs p=0.95 — confidence lowers it (i13 ln via series): live demo
HISTORY & CREDIT Shannon · the log loss
“Cross-entropy and accuracy measure the same thing.” — accuracy only counts right/wrong; cross-entropy grades confidence, punishing a confident mistake far more. cited
surprise · −ln(ptrue) — the information cost of the truth under the model. the pairing · with softmax the gradient is just p − y — clean, stable. Shannon · cross-entropy from information theory, the log loss of classification.
Confidence, graded by surprise. loss
RECOMMEND FOR I-13 the loss, on the compiler
On i-13, −ln(0.665) = 0.408, but −ln(0.95) = 0.051 — being confident and right costs almost nothing:
$ i13 run nn_cross-entropy.i13
RUN OK · 1060 step(s) · call depth 23
ce = 0.408 -- -ln(0.665)
ce_low = 0.051 -- -ln(0.95), confident+right
lower_when_confident = 1
Recommend as a NULL — an information measure (B39). The cross-entropy is a pinned function of the prediction and label; every correct computation agrees. NULL — the loss that punishes confident wrongness.