One attention is one point of view. Split the model dimension into h parallel heads and each looks in a different subspace at once — the transformer reads several relationships in a single pass. The construction is runnable: h heads of size d/h concatenate back to exactly d, each head's weights are a real softmax, and the whole thing collapses to one head when h=1. Down the center, data flows: the input goes in, the heads attend, the projected result comes out. The blue team builds and defends it; the red team tries to break it.
source Vaswani et al., Attention Is All You Need (2017) — arxiv.org/abs/1706.03762. Multi-head attention lets a model “jointly attend to information from different representation subspaces” (Vaswani et al.). Rendered, not quoted.
Multi-head attention is five moves, each checkable:
M1 split model dim d into h heads of size d/h — they concatenate back to exactly d. M2 per head, project X to Q,K,V and take scaled dot-product attention; every row of the weight matrix is a softmax (sums to 1). M3 concatenate the h head outputs, then apply one output projection WO — equal to summing each head projected by its own block of WO. M4 with h=1 it reduces exactly to single-head attention. M5 different heads use different weight slices, so they attend to different patterns — they are independent.
Live invariant check for the current config:
| invariant | holds |
|---|
A single scaled dot-product attention mixes tokens along one learned similarity. Multi-head is that same operation run in h parallel subspaces — the transformer attending to several relationships at once, then folding them back with WO.
The parent sphere is one head; this sphere is h of them side by side. Set h=1 here and the two are literally the same computation — the reduction is a self-check, not a claim. Each sphere is the next one's premise.
The blue team's live check: re-run all five invariants over h ∈ {1,2,4} and confirm them against exact algebra. If red ties the heads together (window 6), invariant M5 breaks and this badge goes red.
Feed a small fixed input X of n=3 tokens × d=4 features (fixed-seed, reproducible). Three learned matrices per head turn each token into a query, a key, and a value:
| symbol | shape | role |
|---|---|---|
| X | n × d | the tokens (input) |
| WQ,WK,WV | d × d | query / key / value maps |
| head i slice | d × d/h | that head's subspace |
| WO | d × d | output projection |
Each head reads a different column-block of the projections — that is why the heads can disagree. This is what you feed the panel below.
Concatenated & output-projected result (n × d):
Switch h — every matrix is computed live from the fixed weights: softmax per head, concat, then one WO. Nothing is looked up.
What the machine produces, proven for h ∈ {1,2,4}: h·(d/h)=d exactly; every head's weight rows sum to 1; concat·WO equals Σ headi·WOi; h=1 reduces to single-head attention; and distinct heads give distinct attention patterns. Five invariants, all exact.
The blue team's witness (left) re-checks these live; the red team (right) tries to make one fail.
And attention is quadratic: O(n²) in sequence length — the real bottleneck, and the reason sparse / linear-attention variants exist. Per-head weights are also not reliable explanations of the model's decision (Jain & Wallace, 2019). The engine proves the algebra, not that any single head "means" something.
"Each head learns a distinct, interpretable role." Cut. Many heads are redundant and prunable (Michel 2019); only some specialize (Voita 2019). The engine only asserts heads can differ, not that each is meaningful.
"Attention weights show what the model uses." Cut. Attention is not explanation (Jain & Wallace 2019). Kept: each weight row is a genuine probability distribution over positions — the engine verifies it sums to 1.
"More heads is strictly better." Kept, corrected. Splitting d into h keeps total width fixed at d; past a point extra heads add cost without accuracy — the paper's own ablation shows returns diminish.
The red team's move: tie every head to head 0's weight slice, so the heads are no longer independent. The blue team's witness (window 7) is watching invariant M5.
Tie the heads and every head produces the same attention pattern — the "heads differ" invariant fails, the witness recomputes, disagrees, and turns red. Softmax and the concat identity still hold; only independence breaks. The attack is real and it is caught.