How many bits does a message really need? Shannon proved there is a floor — H = −Σ p·log₂p bits per symbol — and that a symbol code can get within one bit of it. Down the center, data flows: a symbol-frequency table goes in, the engine measures the entropy and builds the optimal Huffman code, and the proven length comes out. The blue team builds and defends it; the red team tries to break it.
source C. E. Shannon, A Mathematical Theory of Communication, Bell System Technical Journal (1948) — archive.org/details/ost-engineering-shannon1948. Rendered, not quoted.
Two moves, both mechanical:
1 · measure. Turn counts into probabilities pᵢ = cᵢ/N. The entropy is the average surprise, H = −Σ pᵢ log₂ pᵢ — a certain outcome (p=1) adds 0; a fair choice of 2ⁿ options adds n bits. It is the theoretical floor: no code beats it on average.
2 · code. Huffman's rule: give each symbol a leaf weighted by its count, then repeatedly merge the two lightest nodes into one. Rare symbols sink deep (long codes), common ones stay shallow (short codes). The result is a prefix-free code with the shortest possible average length L.
The theorem the machine enforces: H ≤ L < H+1. You can always get within one bit of the floor — and never below it.
Boole gave us a variable that is only 0 or 1 — the algebra of the previous sphere. A logic gate. But Boole's 1 and 0 carried no weight: nothing said one was rarer, or more informative, than the other.
Shannon hands that switch a probability. The moment a bit has a distribution, it has a size — and the whole theory of channels, capacity, and compression opens. That single addition — a measure on the bit — is the door from logic to information. Each sphere is the next one's premise.
The blue team's live check: recompute the anchor entropies from scratch — uniform-4 must be 2.0, a fair coin 1.0, a certain outcome 0 — and confirm the current code obeys H ≤ L < H+1. If red tampers with the log base, this badge is where it shows.
The only input is a source: an alphabet of symbols and how often each occurs. That is all entropy needs — not the meaning of the message, only the shape of its distribution.
A skew makes the source predictable and cheap to encode; a flat distribution is maximally uncertain and costs the most. Edit the counts in the panel below and feed a different source into the engine.
| if the source is… | its entropy H is… |
|---|---|
| one certain symbol | 0 bits |
| a fair coin {½,½} | 1 bit |
| 4 equally likely | 2 bits |
| k equally likely | log₂ k bits (the max) |
Every count is editable. H, the Huffman tree, and L are recomputed on each keystroke — never looked up.
| sym | count | p | −log₂p | code | len |
|---|
What the machine produces, proven: the entropy floor H, the optimal symbol-code length L, and the guarantee that they are trapped together — H ≤ L < H+1. The gap L−H is the redundancy you pay for using whole bits per symbol.
The blue team's witness (left) confirms the anchors live; the red team (right) tries to move the floor.
And Huffman is only optimal among codes that spend whole bits per symbol. Arithmetic and ANS coders spend fractional bits and beat it — approaching H even when one symbol has probability 0.9 (where Huffman is stuck at 1 bit but H ≈ 0.47). The floor is real; this particular code is not the last word.
"Entropy is the information in a message." Cut. It is the average bits per symbol of the source. A single message's meaning is outside the theory entirely — Shannon says so on page one.
"Huffman gives optimal compression." Cut. Optimal only among integer-length prefix codes for a known model. Arithmetic coding does better; the engine shows the L−H gap it leaves.
"Shannon entropy is thermodynamic entropy." Kept, corrected. The formula matches Boltzmann's and von Neumann suggested the name — but this is a measure on a probability distribution, not on heat.
The red team's move: swap the logarithm base — compute H in log₁₀ (decimal digits) instead of log₂ (bits). Everything still "runs", but the floor silently drops.
In decimal units the uniform-4 source measures 0.602, not 2.0 — a genuine change to a computed value. The witness (window 7) recomputes the anchors, sees 2.0 is gone, and turns red. Nothing is faked; the attack is real and it is caught.