Cross-Entropy Loss — Intelligence I

information theory · frontiers · intelligence 1 of 4

Ask how a neural network "learns" and the honest answer is startlingly old: it is driven to reduce a number measured in bits. That number — cross-entropy — is Shannon's idea wearing a new hat, and it is the engine behind nearly every model trained today, this one included. This pamphlet is that loss.

What it is

The objective

The quantity training pushes down, step by step.

the target

The gap

How far the model's predicted distribution sits from the truth.

distance

= KL divergence

Minimizing it is minimizing the relative entropy of Book 2.

the link

In bits

Its units are Shannon's units — bits (or nats) of mismatch.

the scale

The idea

The number that gets minimized

Training adjusts a model to make the cross-entropy between its predictions and the true labels as small as possible.

form L = −Σ yᵢ log ŷᵢ

so "learning" becomes a concrete optimization target.

+1 the true label puts all its weight on the right answer, so the loss is just −log(probability the model gave it).

Measuring a mismatch

Cross-entropy measures how far the model's predicted distribution is from the true one.

reads low when confident and right, high when confident and wrong

so it scores both accuracy and calibration at once.

+1 being confidently wrong is punished hardest — the loss shoots up as the right answer's probability nears zero.

It is KL divergence

Minimizing cross-entropy is mathematically the same as minimizing the KL divergence from truth to model.

link cross-entropy = entropy + KL divergence

so Book 2's relative entropy is the literal training signal.

+1 since the data's own entropy is fixed, pushing cross-entropy down is pushing KL divergence down.

Counted in bits

Its units are Shannon's — bits (base 2) or nats (base e) of mismatch.

scale the same units as entropy

so model error is denominated in information.

+1 a loss of 1 bit means the model is, on average, one fair-coin-flip's worth of surprised — pamphlet 6's idea.

Why it's everywhere

Maximum likelihood, in disguise

Minimizing cross-entropy is the same as making the training data most probable under the model.

equivalence to maximum-likelihood estimation

so it rests on a century of solid statistics.

+1 this is why it feels principled, not arbitrary — it's the classic "fit by likelihood," renamed.

It trains language models

An LLM is trained to predict the next token by minimizing cross-entropy over oceans of text.

task next-token prediction

so the model that writes this was shaped by exactly this loss.

+1 billions of tiny "reduce your surprise at the next word" nudges add up to fluent language.

Well-behaved to optimize

Paired with a softmax output, its gradients are clean — which is why training actually converges.

bonus simple, stable gradients

so it's practical, not just theoretically tidy.

+1 the gradient reduces to "predicted minus true" — elegantly simple, and a big reason it won out.

The core of it

Cross-entropy loss is the quantity nearly every classifier and language model is trained to minimize.
It measures the gap between the model's predicted distribution and the truth — low when confidently right, high when confidently wrong.
Minimizing it is mathematically minimizing the KL divergence (Book 2's relative entropy), and is equivalent to maximum likelihood.
Its units are Shannon's bits — model error denominated in information.
It trains language models via next-token prediction, and has clean gradients that make training converge — which is why the model writing this was shaped by it.

Settled vs. caveat

Settled: the definition, the KL-divergence and maximum-likelihood equivalences, and its role in training are standard, textbook machine learning.
Low loss means good predictive fit, not understanding — the Book 0 caution (information ≠ meaning) applies squarely here.
It optimizes fit to the training distribution; a low loss can still mean memorization, bias, or failure on new data — the metric isn't the whole story.
The formula is the standard classification case; real systems add regularization, weighting, and tricks — the card is the clean core, not the full recipe.
First of four intelligence pamphlets in this eight-part frontier set.

information theory · intelligence frontier · pamphlet 5 of 8 · cross-entropy loss — learning as reducing bits of mismatch