← TTU1 · Transformer Tech Universe · exhibit 5 · the audit-fill

The Lineage

the ideas behind the transformer · ranked by year introduced · 1943 → 2024 · verified online

An audit of this universe's mini-corpus turned up an honest crack — it told the story as if attention began in 2017. It didn't. This is the corrected, web-checked lineage: 37 ideas in order of the year they were introduced, each with the paper and a source link. The cracks are named first; the timeline fills them. Two ⬡ entries (PySvelte 2021, Toy Models of Superposition 2022) are lineage data folded in from Anthropic's early public git.

The Audit — cracks found & filled

what was missing or mis-dated in the mini-corpus, and how it's fixed

⛒ Attention was dated to 2017

TTU1 centered “Attention Is All You Need” (2017) as if attention began there.

FIXED — attention was introduced in 2014 by Bahdanau, Cho & Bengio for translation; 2017 made it work without recurrence (the transformer). Attention's triumph, not its birth.

⛒ No pre-history

The run-up was missing — what the transformer replaced and was built from.

FILLED — the neuron (1943), perceptron (1958), backprop (1986), LSTM (1997) the RNN it replaced, word2vec, seq2seq, and the structural parts it's literally made of (residuals 2015, LayerNorm 2016, GELU, BPE, Adam, dropout).

⛒ No post-2017 lineage

TTU1 had RoPE, scaling-cost, monosemanticity scattered, but no through-line.

FILLED — BERT/GPT (2018) → GPT-2 (2019) → scaling laws + GPT-3 (2020) → RoPE (2021) → CoT + RLHF + Chinchilla + FlashAttention + ChatGPT (2022) → GPT-4 + LLaMA + monosemanticity (2023) → reasoning models (2024).

⛒ MoE's date

Mixture-of-Experts wasn't placed.

FILLED — MoE (2017) actually predates the transformer paper by 5 months (Jan vs June 2017), same lead author (Shazeer).

Foundations 1943–1997

1943

The Artificial Neuron NEW — filled

McCulloch & Pitts

a neuron as a threshold logic unit — neural nets as logical computation

1958

The Perceptron NEW — filled

Rosenblatt

a trainable single-layer classifier with learnable weights

1986

Backpropagation, popularized NEW — filled

Rumelhart, Hinton & Williams

training multi-layer nets by back-propagating errors; learned hidden representations

1997

LSTM NEW — filled

Hochreiter & Schmidhuber

a gated recurrent cell that beats vanishing gradients — the RNN the transformer later replaced

The run-up 2013–2016

2013

word2vec in TTU1

Mikolov, Chen, Corrado & Dean

fast dense word embeddings; vector arithmetic on meaning (king−man+woman≈queen)

2014

Adam optimizer NEW — filled

Kingma & Ba

the default optimizer (as AdamW) for training essentially every LLM since

2014

Dropout NEW — filled

Srivastava, Hinton et al.

the core regularizer used in early deep nets and the original transformer

2014

seq2seq NEW — filled

Sutskever, Vinyals & Le

an encoder–decoder LSTM mapping variable-length input → output

2014

THE ATTENTION MECHANISM in TTU1

Bahdanau, Cho & Bengio

attention itself — additive 'Bahdanau' attention for translation. ★ This predates the transformer by 3 years.

2015

BPE tokenization (for NLP) in TTU1

Sennrich, Haddow & Birch

subword tokenization — the input representation GPT/BERT/LLaMA all build on

2015

Luong attention in TTU1

Luong, Pham & Manning

multiplicative / dot-product attention (global vs local) — the form the transformer uses

2015

ResNet · residual connections in TTU1

He, Zhang, Ren & Sun

the residual skip connection — a literal component of every transformer block

2016

GELU activation NEW — filled

Hendrycks & Gimpel

the activation in BERT/GPT feed-forward layers

2016

Layer Normalization in TTU1

Ba, Kiros & Hinton

the normalization inside transformer blocks (pre-/post-LN)

The transformer 2017–2019

2017

Mixture-of-Experts (sparse-gated) NEW — filled

Shazeer et al.

sparse gating over thousands of experts — the modern MoE (Mixtral, etc.). ★ Predates AIAYN by 5 months.

2017

ATTENTION IS ALL YOU NEED in TTU1

Vaswani et al.

the Transformer — self-attention WITHOUT recurrence/convolution; multi-head attention. Attention's triumph, not its birth.

2018

ULMFiT NEW — filled

Howard & Ruder

transfer learning for NLP: pretrain a language model, then fine-tune

2018

BERT in TTU1

Devlin et al. (Google)

bidirectional masked-LM pretraining — the encoder stack that swept NLP

2018

ELMo NEW — filled

Peters et al.

deep contextualized word embeddings from a bidirectional LM

2018

GPT-1 in TTU1

Radford et al. (OpenAI)

the decoder-only transformer: generative pretrain → fine-tune

2019

Multi-Query Attention / the KV cache in TTU1

Shazeer

caching past keys/values for fast autoregressive decode, and shrinking its memory cost — the inference economy David's 'token tax' meters

2019

GPT-2 in TTU1

Radford et al. (OpenAI)

a 1.5B-param LM with strong zero-shot; the staged-release moment

The scaling era 2020–2022

2020

Scaling Laws in TTU1

Kaplan et al. (OpenAI)

loss falls as a power law in params/data/compute — the thesis that bigger works

2020

GPT-3 in TTU1

Brown et al. (OpenAI)

175B params; in-context / few-shot learning — scale becomes capability

2021

RoPE (Rotary Position Embedding) in TTU1

Su et al.

relative position by rotating Q/K — the positional encoding of most modern LLMs

2021

PySvelte NEW — filled

Anthropic

lineage data ⬡ the org's first public repo (Dec 2021) — a Python↔Svelte bridge for interpretability visualizations, shipping AttentionMulti: attention weights rendered as a readable map. Tooling for looking in.

2022

Chain-of-Thought prompting NEW — filled

Wei et al. (Google)

make the model show intermediate reasoning steps; complex reasoning jumps

2022

InstructGPT · RLHF in TTU1

Ouyang et al. (OpenAI)

alignment by human feedback — a 1.3B model preferred over 175B GPT-3

2022

Chinchilla · compute-optimal scaling NEW — filled

Hoffmann et al. (DeepMind)

for a compute budget, train a smaller model on more data — corrects Kaplan

2022

FlashAttention in TTU1

Dao et al.

IO-aware exact attention — big speed/memory wins that made long context affordable

2022

Toy Models of Superposition NEW — filled

Anthropic

lineage data ⬡ how a network packs MORE features than it has neurons by overlaying them — the grammar of how meaning is encoded in the weights, and the precursor to Towards Monosemanticity.

2022

ChatGPT launches NEW — filled

OpenAI

productized RLHF chat (Nov 30) — the consumer inflection point

The frontier 2023–2024

2023

Towards Monosemanticity in TTU1

Anthropic

sparse autoencoders pull interpretable FEATURES out of a model — looking in

2023

LLaMA in TTU1

Touvron et al. (Meta)

efficient open-weight models — 13B rivals GPT-3 175B; the open-LLM wave

2023

GPT-4 NEW — filled

OpenAI

a large multimodal model — human-level on many professional exams

2024

Scaling Monosemanticity in TTU1

Anthropic

SAE interpretability scaled to a production model (Claude 3 Sonnet)

2024

Reasoning models (o1) NEW — filled

OpenAI

RL-trained inference-time chain-of-thought — spend compute at test time to think

Every year was verified online (arXiv, the original papers, Wikipedia) before listing — see each entry's ↗ source. ★ marks the two that rewrite the common story: attention is 2014 (Bahdanau), not 2017, and Mixture-of-Experts (2017) predates the transformer paper by five months (same lead author, Noam Shazeer). Items tagged in TTU1 were already somewhere in the universe; NEW — filled are the cracks this exhibit closes. The frontier years move fast; this is current to mid-2026.