One attention can only look one way. Multi-head attention runs several attentions in parallel, each on a lower-dimensional slice of the space, then concatenates the results. One head can track syntax while another tracks coreference — different relationships in different subspaces, all in a single layer. Split the model dimension across h heads, attend, and stitch back to the full width.
THE TECHNIQUE h heads on d/h subspaces, concatenated
The demo splits d=8 across h=2 heads (each dim 4) and concatenates back to 8: live demo
HISTORY & CREDIT Vaswani et al. · 2017
“More heads always help.” — many heads are prunable with little loss; a few do most of the work. cited
the split · divide the d-dim space into h subspaces of d/h. attend & stitch · each head attends in its subspace; concatenate back to d. 2017 · Vaswani et al. — different relations, different heads, one layer.
Several viewpoints, in one pass. architecture
RECOMMEND FOR I-13 the split, on the compiler
On i-13, d=8 across h=2 heads gives head_dim 4, and concat restores the full d=8:
$ i13 run n2_multi-head-attention.i13
RUN OK · 22 step(s)
head_dim = 4 (d=8 / h=2)
total = 8 = d concat_restores = 1
Recommend as a NULL — a pinned decomposition (B39). The head split/concat is a fixed reshape of attention (already a pinned function). NULL — many attentions at once.