Perplexity turns entropy into a number you can feel: PPL = e^H, the ‘effective number of equally-likely tokens’ the model is choosing between. A confident model has perplexity near 1 (one real option); a lost one over a size-n vocab has perplexity n. It’s the standard yardstick for how surprised a language model is. Slide to see it swing.
Perplexity PPL = e^H(p) = e^(−Σp log p) is the exponential of the entropy — the ‘effective branching factor’ of the next-token distribution. PPL=1 when the model is certain (one token), and PPL=n when it is uniform over n tokens (maximally perplexed). On held-out text it’s computed as e^(mean cross-entropy) and is THE standard measure of language-model quality: lower means less surprised. A fail-loud self-check throws unless a uniform distribution over n gives PPL=n. ◆ real information theory, node-verified.
Here PPL is the exponential of a single step’s entropy; corpus perplexity averages cross-entropy over many tokens. The 1…n range and e^H identity are exact. Lower PPL is not automatically ‘better output’ — only less surprised.