Reaching the Limit

enihundua · a concept series · book no. 2 · reaching the limit

After 45 years stalled halfway, the wall fell almost overnight — and the codes that broke it now run every phone on Earth. Meanwhile Shannon's idea was leaking out of engineering entirely, into biology, statistics, and economics. This book is the breakthrough, and the great spreading-out.

What happened

Turbo codes

1993: the first practical codes to closely approach the Shannon limit.

the breakthrough

LDPC codes

Invented 1960, ignored, rediscovered in the '90s — now everywhere.

the comeback

Within ~1 dB

Modern codes hug the limit so closely there's little room left.

near-perfect

The spread

Entropy and information escape into other sciences entirely.

beyond comms

Closing the gap

Turbo codes (1993)

Berrou and Glavieux stunned a conference with codes that nearly hit the Shannon limit.

who Claude Berrou & Alain Glavieux, France

so the 45-year wall fell — capacity was suddenly within reach.

+1 their claimed performance was so good that many experts first assumed it was an error.

The trick: iteration

Two simple decoders pass guesses back and forth, refining each other — "turbo" like an engine's feedback.

idea iterative, probabilistic decoding

so near-optimal decoding became practical, not just theoretical.

+1 turbo codes power 3G, 4G, and 5G — the breakthrough is in the phone in your pocket.

LDPC: ahead of its time

Robert Gallager invented low-density parity-check codes in his 1960 MIT PhD — then they were shelved.

why ignored 1960s hardware couldn't decode them

so a near-optimal idea sat unused for decades.

+1 Gallager lived to see them everywhere — winning prizes for work he'd done over 30 years earlier.

LDPC rediscovered (~1996)

Amid the turbo-code excitement, David MacKay and others realized Gallager's codes also nearly reach the limit.

bonus LDPC is unpatented — free to use

so it spread into Wi-Fi, DVB-S2, 5G, and Blu-ray.

+1 being patent-free is exactly why open standards favored LDPC over the patented turbo codes.

Spreading out

Algorithmic information

Kolmogorov (and others) asked: what's the shortest program that could generate this data?

idea complexity = length of the shortest description

so "information" gained a second, computational meaning.

+1 it links Shannon's entropy to the very notion of randomness — random means incompressible.

Relative entropy (KL divergence)

Kullback and Leibler measured how far one probability distribution sits from another.

tool KL divergence

so statistics gained an information-based notion of "difference."

+1 this single quantity becomes the bridge to modern machine learning in Book 3.

Mutual information

A measure of how much knowing one thing tells you about another.

idea shared information between variables

so fields could quantify dependence without assuming a formula.

+1 neuroscientists use it to ask how much a neuron's firing "tells" about a stimulus.

The Bandwagon warning

As information theory spread into psychology, biology, and economics, Shannon urged caution.

essay "The Bandwagon" (1956)

so he reminded everyone it wasn't a cure-all for every field.

+1 a founder reining in his own hype — the careful applications endured, the overreaches faded.

The story so far

Turbo codes (1993), by Berrou and Glavieux, were the first practical codes to closely approach the Shannon limit, using iterative decoding — and now run 3G/4G/5G.
LDPC codes, invented by Gallager in 1960 but ignored, were rediscovered around 1996; near-optimal and patent-free, they spread into Wi-Fi, 5G, and Blu-ray.
Modern codes now get within about 1 dB of the limit — so close that research has shifted to making decoding faster and more adaptive.
Meanwhile the idea spread beyond communication: algorithmic (Kolmogorov) complexity, KL divergence, and mutual information reached statistics, neuroscience, and economics.
Shannon's own "Bandwagon" (1956) warned against overreach — and the disciplined uses are what lasted.

How we know — and the honest caveats

The turbo/LDPC story, dates, and people are well-documented (MIT's own histories, the original 1993 ICC paper, Gallager's 1960 thesis).
"Within ~1 dB" and "closely approach" are real but channel-dependent — performance varies with code length, noise, and conditions; no practical code reaches the limit exactly.
Turbo vs. LDPC isn't a simple winner — they're competing families with different trade-offs; standards pick by latency, complexity, and licensing.
The "spread into other fields" is genuine but uneven — some applications are rigorous, others were the very overreach Shannon cautioned against. Treat sweeping "information theory explains X" claims with care.
Book 3 carries the idea to its current frontiers: quantum information and modern AI.

information theory · book no. 2 · the wall falls, the idea escapes · reaching the limit & spreading out (1993–)