The mechanism that made transformers: for each query, score it against every key by dot product, scale by √d, softmax the scores into weights, and take that weighted sum of the values — softmax(QKᵀ/√d)·V. It lets a token pull information from any other token in one step, no matter how far apart. “Attention is all you need” — the whole architecture behind modern language models.
THE TECHNIQUE softmax(QKᵀ / √d) · V
The demo scores a query against two keys, scales by √d, and softmaxes — attention lands on the matching key: live demo
HISTORY & CREDIT Bahdanau 2014 · Vaswani 2017
“Attention weights explain what the model is thinking.” — they show where information is pulled from, but are a debated, imperfect explanation of the model’s reasoning. cited
score · query · key by dot product; scale by √d to keep the variance sane. weight & mix · softmax the scores, take the weighted sum of values. 2017 · Vaswani et al., “Attention Is All You Need” (after Bahdanau 2014) — the transformer.
Any token, reaching any other, in one step. architecture
RECOMMEND FOR I-13 the weights, on the compiler
On i-13, a query matching key 1 (score 1) vs key 2 (score 0), scaled and softmaxed, puts 0.67 of the weight on key 1:
$ i13 run nn_attention.i13
RUN OK · 1310 step(s) · call depth 31
scaled score = 0.707 (1/√2)
w = [0.670, 0.330] -- attends to the matching key
focuses_on_match = 1 weights_sum_1 = 1
Recommend as the KEEPER SHOT. Attention routes information by content, no fixed wiring — the most structural card here. But softmax(QKᵀ/√d)V is a pinned function of Q,K,V; every correct implementation returns the same weights (B39). The panel rules; expected NULL — a fixed computation, not a same-function DOF.