Attention looks at all tokens at once — it has NO built-in sense of order, so ‘dog bites man’ and ‘man bites dog’ would look identical. The fix: stamp each position with a unique fingerprint made of sines and cosines at many frequencies, and add it to the token. Nearby positions get similar stamps; far ones differ; and a fixed rotation shifts any stamp to its neighbour’s. Slide the position and watch its wave-signature.
Self-attention is permutation-equivariant — shuffle the inputs and the outputs just shuffle — so a transformer needs POSITION added explicitly. The sinusoidal encoding gives position pos a vector PE(pos, 2i) = sin(pos / 10000^(2i/d)), PE(pos, 2i+1) = cos(…): a bank of sinusoids from high to very low frequency, so every position gets a distinct fingerprint and nearby positions get similar ones. Crucially PE(pos+k) is a FIXED linear (rotation) function of PE(pos), letting attention learn relative offsets. A fail-loud self-check throws unless different positions get different vectors. ◆ real ML mechanism, node-verified.
The original sinusoidal encoding (the exact per-position uniqueness and relative-shift-by-rotation); learned and rotary (RoPE) encodings differ in form — the ‘attention needs a position stamp’ necessity and the sinusoidal structure are exact.