Surprise in Bits — Intelligence II

information theory · frontiers · intelligence 2 of 4

How do you grade a mind — human or machine — on a prediction? Shannon's answer: measure its surprise. A likely event that happens is no news; an unlikely one carries many bits. Stack that up and you get the standard yardstick for every language model. This pamphlet is surprise as a number.

The chain

Surprisal

One event's surprise: −log of its probability.

−log p

Entropy

The average surprise over all outcomes — Shannon's H.

the mean

Perplexity

Entropy made friendly — "how many equally-likely choices?"

the score

Bits per token

The benchmark for how well a model predicts language.

the gauge

The idea

Surprise = −log p

The less likely an event, the more "surprise" — and the more information — it carries when it happens.

name self-information, or surprisal

so information and surprise are literally the same quantity.

+1 a certain event (p = 1) carries zero bits — no surprise, no news. The math matches the intuition exactly.

Rare events cost more

A one-in-a-thousand outcome carries far more bits than a coin flip.

scale halving the probability adds one bit

so surprise grows as events get rarer.

+1 this is why compression gives rare symbols longer codes — they genuinely carry more information.

Entropy is average surprise

Shannon's H is just the expected surprisal over all possible outcomes.

link H = average of −log p

so entropy and "typical surprise" are one idea.

+1 it reframes Book 0's H in one line: entropy is how surprised you should expect to be, on average.

A model's surprise is its loss

Cross-entropy (pamphlet 5) is exactly the model's average surprise at the true answers.

link loss = average surprisal under the model

so "minimize loss" means "be less surprised by reality."

+1 a perfectly-predicting model is never surprised — its loss bottoms out at the data's own entropy.

The standard yardstick

Perplexity

The headline language-model score — entropy exponentiated into an intuitive number.

reads as "effective number of equally-likely next words"

so lower perplexity = a model less confused by text.

+1 perplexity 1 would mean perfect prediction; in practice it's a direct stand-in for bits of surprise.

Bits per character

Models and even humans are benchmarked by how many bits they need per character of text.

echo Shannon's own 1950 experiments on English

so the yardstick runs straight back to Shannon himself.

+1 Shannon estimated English at ~1 bit per letter by having people guess the next one — a 1950 perplexity test.

It connects to compression

Fewer bits of surprise per symbol means the text compresses smaller — the bridge to pamphlet 7.

link low surprise ⇄ high compressibility

so prediction quality and compression are two readings of one gauge.

+1 a model's bits-per-token is the size it could compress that text to — prediction and compression, one number.

The core of it

Surprisal (−log p) measures one event's surprise — and is identical to the information it carries. Rare events cost more bits.
Entropy (H) is just the average surprise over all outcomes — Book 0's measure, seen from a new angle.
A model's cross-entropy loss is its average surprise at the true answers — "minimize loss" means "be less surprised by reality."
Perplexity turns that into an intuitive score ("effective number of choices"); bits-per-character is the benchmark — echoing Shannon's own 1950 experiments.
Fewer bits of surprise per symbol means better compression — the bridge to pamphlet 7.

Settled vs. caveat

Settled: surprisal, entropy-as-average-surprise, and perplexity are standard information theory and NLP — Shannon's 1950 English-entropy work is the original.
Low surprise measures predictive fit, not comprehension — a model can be unsurprised by text it doesn't "understand" in any deeper sense. (Book 0's caution, again.)
Perplexity numbers are only comparable under the same tokenization and data — a lower score isn't automatically a "smarter" model across different setups.
"Surprise" here is the technical sense (−log p), not the emotional one — a precise quantity, not a feeling.
Second of four intelligence pamphlets in this eight-part frontier set.

information theory · intelligence frontier · pamphlet 6 of 8 · surprise in bits — how you grade a prediction