Information Theory

enihundua · a concept series · book no. 0 · the idea

In one 1948 paper, Shannon turned a vague word into a science. He proved information can be measured, compressed to a hard floor, and sent error-free through noise — then left a famous challenge unanswered. This book is the idea at its birth. The next three trace how the world spent 75 years rising to it.

Before & at the start

The precursors

Nyquist and Hartley (1920s) first asked how fast a line could carry "intelligence."

the runway

The bit

The atom of information — one answer to one yes/no question.

the unit

Entropy (H)

The true, irreducible amount of information a source produces.

the measure

Capacity (C)

The top speed at which a channel can carry information error-free.

the limit

The 1948 breakthrough

Standing on Nyquist & Hartley

In the 1920s, Bell engineers Harry Nyquist and Ralph Hartley studied how much "intelligence" a channel could carry.

when the 1920s, at Bell Labs

so Shannon's theory built directly on their earlier ideas.

+1 Hartley even used a log measure of information — Shannon credited this groundwork explicitly in 1948.

Information becomes a quantity

Shannon defined the bit and showed any message could be measured in bits.

idea a common currency for all communication

so text, sound, and image became the same kind of thing.

+1 crucially he set meaning aside — engineering the pipe, not the message — which is what made it a science.

Entropy: the floor

He defined entropy H — the smallest number of bits that can represent a source.

result the source coding theorem

so compression has a hard, provable limit.

+1 a predictable source has low entropy — exactly why ordinary text squeezes down so well.

Capacity: the ceiling

He proved every noisy channel has a capacity C, below which error-free sending is possible.

result the noisy-channel coding theorem

so reliable communication through noise became a law.

+1 counterintuitive then: any noise can be beaten, if you add the right redundancy and stay under C.

The gauntlet

Good codes must exist…

Shannon proved that codes approaching the capacity limit are guaranteed to exist.

claim near-perfect codes are possible

so engineers knew the prize was really there.

+1 his proof was non-constructive — he showed they exist without exhibiting a single one.

…but he didn't show how

He left the actual construction of those codes to future generations.

the challenge "throwing down the gauntlet"

so a 45-year search for practical codes began.

+1 for decades, real codes stalled at the "computational cutoff rate" — often only halfway to the limit.

The Shannon limit

The maximum reliable rate became the benchmark the whole field would chase.

name the Shannon limit

so progress could be measured as a "gap to capacity."

+1 Books 1–2 trace the long climb; modern codes finally get within about 1 dB of it.

A new science is born

The paper founded an entire discipline overnight: information theory.

work "A Mathematical Theory of Communication"

so communication had, for the first time, fundamental laws.

+1 Quanta called it as if he "discovered the universe's laws of communication" rather than invented them.

The core idea

Building on Nyquist and Hartley, Shannon's 1948 paper made information a measurable quantity, counted in bits.
Entropy (H) sets a hard floor on compression: no lossless code can shrink a source below it.
Channel capacity (C) sets a ceiling: below it, communication can be made error-free even through noise, with enough clever redundancy.
He proved good codes exist near that ceiling — but didn't build them, leaving a challenge that took decades.
That maximum reliable rate is the Shannon limit, the north star for everything that followed.

How we know — and what's settled

The 1948 results are settled mathematics — proven theorems, taught worldwide, not open questions. What evolved is how to achieve them in practice (Books 1–3).
The theory deliberately measures bits, not meaning. Treating Shannon "information" as the same as knowledge or significance is the most common misreading.
The "gauntlet" framing is real: Shannon's capacity proof was non-constructive, which is precisely why finding practical codes became a decades-long quest.
Dates and credit here are well-documented (Bell Labs records, the original paper); Nyquist and Hartley's priority is acknowledged in Shannon's own text.
This is a concept series, not a biography — for Shannon the man, see his four-book ENIHUNDUA set.

information theory · book no. 0 · the bit, the floor, the ceiling, the challenge · the idea at its birth (1948)