THE POSITIONAL ENCODING — a dart

THE POSITIONAL ENCODING give the transformer a sense of order

Attention treats a sentence as a bag of tokens — it has no built-in notion of which word came first. So the transformer adds a positional code to each token: a set of sine and cosine waves of different frequencies, PE(pos) = sin(pos/10000ᵢ), giving every position a unique, smooth fingerprint. The waves also let the model read relative distances by simple linear combinations.

THE TECHNIQUE PE(pos, 2i) = sin(pos / 10000ᵢ)

The demo shows different positions get distinct sinusoidal codes (i13 sin via Taylor): live demo

HISTORY & CREDIT Vaswani et al. · 2017

“Transformers know word order.” — attention is permutation-equivariant; without positional encoding it cannot tell ‘dog bites man’ from ‘man bites dog.’ cited

the bag · attention alone is order-blind — a set, not a sequence.
the waves · add sinusoids of many frequencies — a unique fingerprint per position, relative distances linearly readable.
2017 · Vaswani et al. — order injected into the transformer.

Order, painted on as waves. architecture

RECOMMEND FOR I-13 the fingerprints, on the compiler

On i-13, positions 0,1,2 get distinct codes (sin 0, 0.841, 0.909) — a deterministic per-position pattern:

$ i13 run n2_positional-encoding.i13 RUN OK · 1469 step(s) pe(0)=0 pe(1)=0.841 pe(2)=0.909 zero_at_start = 1 distinct = 1

Recommend as a NULL — a pinned function of position (B39/B44). The encoding is fixed by the position; a representation (B44). NULL — give the transformer a sense of order.