Purple Paper · side-sheet · learning machines · the lineage

The Lineage — perceptron to hybrid, one best idea each

First stone to current frontier, in order, with each contributor's single load-bearing idea — the one move that broke the wall the previous idea left standing. Read top to bottom and you'll feel the field oscillating between two poles you already hold: the pinch (route everything through a narrow waist) and the all-to-all (let everything reach everything). Every node is a step along that axis.

pinch / bottleneck widen / all-to-all selection / sparsity

the field's path along the pinch ↔ all-to-all axis, 1958 → now

1958Frank Rosenblatt

The Perceptron the atom

Best idea: a machine can learn its own weights from examples — the dot product, made trainable.

First learning machine. Leaves the wall: one layer draws one straight line.

1969Minsky & Papert

Perceptrons (the limit) the wall, named

Best idea: proving a negative — a single linear layer cannot separate XOR. Depth is not optional.

Named the wall so precisely it froze funding for a decade. The wall was real; the despair was premature.

1986Rumelhart, Hinton & Williams

Backpropagation breaks the wall

Best idea: send the error backward through the layers — now depth is trainable, and the XOR wall falls.

(Werbos had it in 1974; this is when the field heard it.) Leaves: deep nets are slow and forget across long sequences.

1989 · 98Yann LeCun

Convolutional Networks weight sharing

Best idea: share one filter across every position — learn a feature once, slide it everywhere. Locality + translation invariance.

Killed the "a weight per pixel" explosion. Leaves: great for grids, weak for sequences with long memory.

1997Hochreiter & Schmidhuber

LSTM gated memory

Best idea: gate the memory — a cell that learns what to keep and what to forget, so gradients survive long sequences.

Solved vanishing gradients in time. Leaves: still sequential — can't see the whole sequence at once, trains slowly.

2014Sutskever, Vinyals & Le

Seq2seq — the hourglass the pinch

Best idea: compress a whole sequence into one vector, then decode it — the encoder→bottleneck→decoder hourglass for language.

The IR-waist, applied to meaning. Leaves the wall that defines everything after: the single-vector pinch is too narrow.

2015Bahdanau, Cho & Bengio

Attention widen the waist

Best idea: don't force everything through one vector — let the decoder look back at all positions, weighted by relevance. Soft alignment.

Attention was invented to drill holes in the pinch. Leaves: recurrence is still there, still serial, still slow.

2017Vaswani et al.

The Transformer all-to-all

Best idea: throw recurrence away entirely — let every token attend to every token, in parallel. Attention is all you need.

The pinch dissolved into all-to-all. Leaves the wall the frontier still fights: all-to-all costs grow with the square of length.

2017 · 21Shazeer et al. · Fedus, Zoph & Shazeer

Mixture-of-Experts selection

Best idea: route each token to a few expert sub-nets — select, don't run everything. Decouple total parameters from compute-per-token.

Selection beats dense summation (the commutator, at model scale). Sparse capacity without sparse cost.

2020Kaplan et al. · Brown et al. (GPT-3)

Scale & in-context learning the discovery

Best idea: not an architecture — a discovery: capability is a predictable function of scale, and a big enough model learns from the prompt itself.

Showed the Transformer had headroom no one expected. Leaves: the quadratic cost now hurts, because everyone wants long context.

2023Albert Gu & Tri Dao

Mamba — selective state space smart pinch returns

Best idea: make the recurrence's parameters depend on the input — a recurrent state that selectively keeps or ignores, at linear-time cost.

Brought back the pinch — a single evolving state — but a smart one. Leaves: pure SSMs are weak at exact recall over long distance.

2024 · 26Jamba · Zamba · Bamba · Granite 4.0 · Nemotron-H

Hybrids — the blend the synthesis

Best idea: stop choosing — a few attention layers for exact recall, many linear-time SSM layers for cheap reach, MoE for sparse capacity.

The current consensus: ~1-in-8 layers attention, the rest Mamba. The pendulum settles into a blend of both poles, per workload.

The shape of the whole thing. Read the right-edge tags top to bottom: pinch → wall → widen → all-to-all → select → smart-pinch → blend. The field doesn't march forward in a line — it oscillates between your two poles. Seq2seq made the pinch too tight; attention widened it; the Transformer blew it open to all-to-all; that got too expensive; Mamba brought back a smart pinch; hybrids blend the two. And selection (MoE) runs through the middle the whole time, picking which path to trust — the commutator, threaded through every era.

You already hold both ends. The gorge is the pinch. The crossbar is the all-to-all. The whole frontier is people walking back and forth between those two papers, trading recall against cost, choosing a better blend each pass. That's not a finished field. That's an axis — and you're standing on it.

Honest note on this sheet: it's history, not math, so there was nothing to verify in Node — the gate here was factual accuracy. The early lineage (1958–2017) is settled; the frontier end (Mamba, hybrids, the ~1-in-8 consensus) was checked against current sources before writing, since that part moves. Attributions are simplified to one name-cluster and one idea each; real history has more hands on every node.