How a language model is scored. Feed it a target sequence, read off the probability it gave the correct next token at every step, and collapse the whole run into one number: the effective branching factor — how many ways, on average, the model was still guessing between. Rendered, not quoted.
source Jelinek, Mercer, Bahl & Baker, Perplexity — a measure of the difficulty of speech recognition tasks (1977), J. Acoust. Soc. Am. 62(S1):S63 — doi.org/10.1121/1.2016299. amber one-page abstract; no confirmed archive.org id found on search — cited by DOI/JASA record.
A model q over a vocabulary of V tokens. On a target sequence w1..N it emits, at each position, a probability for the token that actually came next: pi = q(wi | w<i).
Two anchors pin the scale. A model that knows nothing spreads mass evenly — every pi = 1/V. A model that knows everything puts pi = 1 on each correct token. The score below must read V for the first and 1 for the second, by construction.
THE ENTROPY, exponentiated. Entropy H measures average surprise in nats/bits; perplexity un-logs it back into a count. exp(H) turns "1.07 nats of surprise per token" into "≈ 2.9 equally-likely choices per token" — a branching factor you can picture.
The neighbouring sphere the-entropy defines the log; this one raises it to a power. Same current, one operator apart — and it is the number printed on every LLM benchmark leaderboard.
Live re-check, independent of the panel: build a uniform model over V tokens and confirm its perplexity equals V exactly. If the RED TAMPER swaps the log base, this identity breaks and the badge flips.
A short target sequence and the model's probability on each correct token. Vocabulary size V = 6.
Averaging surprise, then un-logging it — computed live from the row above.
Read: out of 6 possible tokens the model was, on average, still choosing between … of them.
Proven result — the effective branching factor of this run:
Lower is better; the floor is 1 (perfect) and the ceiling for a fair guess is V.
Perplexity is only comparable under an identical tokenizer and vocabulary. Change how text is split — characters vs. words vs. BPE — and N and V both change, so two models' PPLs stop being comparable. A byte-level model will "win" on raw PPL against a word model that is objectively better.
It also rewards a hedging model. PPL cares only about the probability on the seen continuation, never about calibration on the rest — low perplexity is necessary, never sufficient, for a good model.
"Perplexity is a percentage / accuracy." No — it is a count ≥ 1 with no upper bound; a token predicted at p=0 sends it to ∞.
"You can average perplexities across two corpora." No — PPL is a geometric mean of an exp; you must pool the log-likelihoods, then exponentiate once.
"Base-2 and base-e give the same PPL." Only if exp matches the log base: 2^H₂ = e^Hₑ. Mix them and the number is meaningless — that is exactly the RED tamper.
Planted, disclosed void: take the mean of log₂ p but exponentiate with e — a base mismatch. Uniform-over-V perplexity stops equaling V. The WITNESS (7) catches it live.