Before it picks a token, how UNSURE is the model? Shannon entropy H = −Σp log p measures exactly that — zero when one token is certain, maximal (log n) when all are equally likely. It’s the width of the snowstorm the next token has to cross. Slide from certainty to a coin-flip and watch H rise.
Shannon entropy H(p) = −Σᵢ pᵢ log pᵢ quantifies the uncertainty of the next-token distribution: H=0 when one token has all the mass (perfect confidence), and H=log n when the n tokens are uniform (maximal confusion). It is the theoretical floor on the average bits needed to encode the choice, and it underlies temperature (which raises H), typical sampling (which targets it), and confidence-based stopping. A fail-loud self-check throws unless a one-hot distribution gives H=0 and a uniform one gives H=log n. ◆ real information theory, node-verified.
The morph from peaked to uniform is illustrative; H and its 0…log n bounds are exact. High entropy means the model is unsure HERE — not that any particular token is right or wrong.