Prediction is Compression — Intelligence III

information theory · frontiers · intelligence 3 of 4

Here is the idea that ties the whole series into a knot: to predict the next symbol well is to compress the data well — they are, provably, the same skill. It runs from Shannon's bits to Book 2's Kolmogorov complexity, and it's why some argue that learning to compress just is learning. This pamphlet is that equivalence — and its honest limits.

The equivalence

Predict ⇄ compress

A good next-symbol predictor is a good compressor, and vice versa.

two faces

The mechanism

Arithmetic coding turns predictions directly into short codes.

the bridge

The benchmark

The Hutter Prize: compress Wikipedia as a test of intelligence.

the wager

The hypothesis

Maybe intelligence itself is deeply about compression.

the claim

The idea

Two faces of one skill

If you can predict what comes next, you can encode it in few bits — and a good compressor must predict well to do its job.

link prediction quality = compression rate

so the two are mathematically interchangeable.

+1 it follows straight from pamphlet 6: bits of surprise per symbol is the compressed size.

Arithmetic coding is the bridge

Feed a predictor's probabilities into arithmetic coding and you get near-optimal compression automatically.

mechanism probabilities → code lengths

so any predictor can be turned into a compressor directly.

+1 this is the concrete machine behind the slogan — Book 1's arithmetic coding, driven by a model.

Back to Kolmogorov

Book 2's idea — the shortest program that generates the data — is the deepest form of this equivalence.

link compression ≈ finding the shortest description

so understanding data and shrinking it converge.

+1 Solomonoff built a theory of prediction on exactly this: predict by favoring the simplest explanation.

LLMs are compressors

A language model trained to predict text is, by this logic, a powerful lossless compressor of text.

result "language modeling is compression"

so the model writing this is, in a precise sense, an entropy estimator.

+1 recent work showed large models can out-compress purpose-built tools — prediction skill cashing out as bits saved.

The wager & the claim

The Hutter Prize

A standing contest to compress a fixed slice of Wikipedia as small as possible — framed as an intelligence test.

premise better compression = better understanding

so "compress this text" becomes a benchmark for AI.

+1 the bet is explicit: Marcus Hutter argues compressing well requires understanding the content.

"Intelligence is compression"

Some researchers propose that finding compact patterns is the core of intelligence itself.

idea learning = building a shorter description of the world

so Shannon's bits reach all the way to a theory of mind.

+1 it's a beautiful, suggestive idea — and, importantly, a hypothesis, not a settled fact (see the footer).

The series, in a knot

Bit → entropy → surprise → cross-entropy → compression → prediction — one thread, tied off here.

through-line every pamphlet meets in this one

so Shannon's 1948 idea reaches the heart of modern AI.

+1 the last pamphlet (8) closes the loop: the AI built on these ideas is named for the man who began them.

The core of it

Predicting the next symbol well and compressing data well are the same skill — provably, via the bits-of-surprise accounting of pamphlet 6.
Arithmetic coding is the concrete bridge: feed it a predictor's probabilities and it produces near-optimal compression.
At its deepest, this is Kolmogorov complexity (Book 2) — compression as finding the shortest description — and the basis of Solomonoff prediction.
So a language model is a compressor; recent work shows large models can even out-compress dedicated tools.
The Hutter Prize turns this into an AI benchmark, and some argue "intelligence is compression" — a serious, suggestive hypothesis.

Settled vs. hypothesis

Settled: the prediction-compression equivalence is rigorous and standard — it's basic information theory, and the LLM-as-compressor result is demonstrated.
A genuine hypothesis, not a law: "intelligence is compression" is a real research idea that remains contested. It's suggestive and partly formalized (Solomonoff, Hutter), but not proven — treat it as a live claim, not established fact.
Compression captures pattern-finding and prediction, which are part of intelligence — but reasoning, goals, embodiment, and understanding may not reduce to it. Skeptics push exactly here.
"Lossless compression of text" is a precise technical claim; don't over-read it as the model "understanding" in a human sense (Book 0's caution).
Third of four intelligence pamphlets in this eight-part frontier set.

information theory · intelligence frontier · pamphlet 7 of 8 · prediction is compression — the knot that ties the series