The one line that turns attention into a language model. Attention lets every position read every other position — a bidirectional smear of context. Slide a lower-triangular mask under the scores, forcing score[i][j] = −∞ for every future j > i, and the machine can no longer see the token it is trying to predict. Position i now depends only on 0…i. The output at each step is frozen against everything that comes after it — the autoregressive property, the core of every GPT. Down the center, data flows: keys and queries go in, the masked softmax runs, the causal outputs come out. Blue builds and defends the mask; red tears it off.
source Vaswani et al., Attention Is All You Need (2017) — decoder self-attention masking, §3.2.3 — arxiv.org/abs/1706.03762; applied as a pure left-to-right LM by Radford et al., GPT (2018). Rendered, not quoted.
Causality is not a promise; it is four checkable facts about the masked softmax:
C1 strict causality — weight[i][j] is exactly 0 for every future j > i (−∞ → softmax → 0, not merely small). C2 each row still sums to 1 over the positions it is allowed to see. C3 the autoregressive property — output[i] is bit-identical when any future token (index > i) is altered. C4 position 0 attends only to itself.
Live, for the seeded 5-token sequence:
| check | result |
|---|
The attention heads are symmetric by construction: any query may read any key. That is exactly what you cannot allow if the task is predict the next token — the answer would be sitting in the input.
The mask is the single deletion that fixes it: forbid the diagonal's upper half, and a head that once mixed the whole sentence becomes a head that predicts x[i+1] having seen only x[0…i]. Every position becomes a training example at once (teacher forcing), yet none can cheat. That one triangle is the seam between attention and GPT — each sphere is the next one's premise.
The blue team's live guard: recompute the whole masked attention and confirm C1–C4 against the seeded truth. If red tears off the mask (window 6), the future weights go non-zero, early outputs start moving, and this badge flips red.
A single self-attention head over N=5 positions, head dim d=4. From a fixed seed we draw Q, K, V ∈ ℝ5×4, then score[i][j] = (Q[i]·K[j]) / √d. That raw score is bidirectional — everyone sees everyone. The causal mask is the shape below: a cell is allowed only when j ≤ i.
The dark upper triangle is set to −∞ before the softmax. That is the entire intervention — no new weights, no new layer. It is what you feed the panel below.
CAUSAL mode: each row's softmax runs only over j ≤ i. Alter a later token's value V[p] and watch every earlier output stay bit-identical.
Attention weights (row i attends to column j):
| pos | output vector (d=4) | vs base |
|---|
Every number is computed here from Q·K/√d and a live softmax — nothing is looked up. Flip to BIDIRECTIONAL to see the mask's absence leak the future backward.
What the machine proves, per boot: for the seeded head, weight[i][j] = 0 on the whole future triangle, every row sums to 1, position 0 is self-only, and altering token p leaves outputs 0…p−1 bit-identical (C3). That last fact is the property — it is why the same forward pass both trains on and generates a sequence left to right.
The blue witness (left) re-checks these live; the red team (right) removes the mask and this collapses.
And the triangle only closes one door. Causality leaks anywhere the sequence axis is pooled without a mask: a mean/attention pool over all positions, a convolution that reaches right, a normalization computed over the full length, or a padding/label-shift off by one. The mask makes this softmax causal; it cannot make a careless network causal.
"Masking sets future weights to a tiny value." Cut. Future scores are −∞, so exp gives exactly 0 — checked bit-exact in C1. A merely large negative would leak a little probability and break autoregression silently.
"The whole transformer is masked." Cut. Only decoder self-attention is causal. Encoder self-attention and encoder→decoder cross-attention are unmasked — the decoder is allowed to see the whole source.
"The mask makes the model causal." Kept, corrected. It makes one attention layer causal. End-to-end causality is a property you must test (perturb a future token, C3) — not assume from the presence of a triangle.
The red team's move: remove the mask — let every query read every key (full bidirectional attention). Now output[i] mixes in future values, so altering a later token bleeds backward and the autoregressive property (C3) dies. The blue witness (window 7) is watching.
Strip the triangle and the future's weight is no longer 0; earlier outputs start depending on tokens that come after them. The witness recomputes, C1 and C3 fail, and it turns red. Nothing is faked — the leak is real and it is caught.