Now a stack of identical layers goes to work on the sequence of vectors. The engine of each layer is attention: every token looks at every other and pulls in what is relevant, mixing the sequence so each position becomes a blend of its context. Stacked dozens deep, this is where the model's 'thinking' happens — and where the T in GPT comes from.
The Four Natures
each piece emerges by one of four natures
The Idea
the three-beat story
The transformer's core move: at each layer, every token computes how much to attend to every other, then updates itself by pulling in a weighted blend of them. A pronoun reaches back to its noun; a verb gathers its subject and object. Context flows sideways across the whole sequence at once.
Attention runs in parallel heads — different heads track different relationships: syntax, reference, position — and the whole layer repeats, stacked dozens deep. Each pass refines the representations a little, so a token that began as just 'bank' slowly becomes 'bank (river)' or 'bank (money)' from its neighbors.
Information rides a residual stream straight up the stack; each layer reads from it and writes back. By the top, every position's vector has absorbed enough of the whole context to say what comes next. This is the machinery — discovered in detail, not designed — that turns geometry into competence.
Every Token Looks at Every Other
the engine of each layer is attention: a token pulls in a weighted blend of the tokens before it. Hover a token to see what it attends to (thicker arc = more attention); the three colors are three heads tracking different relationships; step the layer to refine. An illustration — the weights are a readable stand-in, not a real model.
The Reckoning
the stage, and the honesty about it
Pipeline · Stage Three
the thread
- The third arrow: a sequence of vectors → a contextualized sequence. The bulk of the model's parameters and compute live right here.
- >Receives embeddings (stage two); its top layer feeds the next-token distribution (the final stage).
Render, Not Invent
honest about the demo
- The live attention above is an illustration — hover a token to see which others it attends to. The weights are a readable stand-in, not a real model's.
- The mechanism — query-key-value attention, multiple heads, the residual stream, stacked layers — is the actual architecture (Vaswani et al., 2017, 'Attention Is All You Need').
Two-Layer Honest
mechanism vs interpretation
- Settled: the architecture, and that attention mixes context across the sequence, are well-established and directly inspectable.
- Open: exactly what each head and layer 'means' is the active science of mechanistic interpretability — read from the weights, not assumed. Cross-links the AI domain's transformer spheres.
The Roster
the parts and concepts of this stage as ACI .agents — each a birth certificate & a nature (11)
