THE POSITIONAL ENCODING

A transformer sees a bag of tokens — order-blind by construction. This instrument builds the sinusoidal positional encoding live and proves the one property that makes it work: every position gets a unique, bounded fingerprint, and a shift by k is a fixed rotation the model can learn once and reuse everywhere. Rendered, not quoted.

source Vaswani et al., Attention Is All You Need (2017) — arxiv.org/abs/1706.03762  ·  §3.5 sinusoidal encoding. arXiv id confirmed by search.

Blue Team · builds & defends
3

THE MODEL

For model width d (even) and integer position pos, split the vector into d/2 pairs. Pair i gets angular frequency ωi = 1 / 100002i/d:

PE(pos, 2i)  = sin(pos · ωi)
PE(pos, 2i+1) = cos(pos · ωi)

Geometric frequencies span from ω=1 (dim 0, fastest) down to a slow crawl — a ruler with many nested tick spacings. Here d=16, checked over 64 positions.

5

THE LINEAGE

Feeds → the-attention. Self-attention is a set operation: permute the inputs and the outputs just permute — it cannot tell position 1 from position 5. This sphere injects order as geometry added to the embeddings, so the order-blind mechanism downstream can finally see sequence.

Order is not stored in attention; it is handed to attention, already shaped.

7

THE WITNESS

Live re-check of the running engine. Confirms bounds and the rotation relation on the current state, and turns red the instant window 6 tampers.

Verdict: checking…

Re-runs invariantsHold() over all 64 positions on every click and every tamper toggle.

The Machine
4

DATA IN in ↓

Positions 0…63 and dimension width d=16. No text, no training — pure index geometry. Two probe positions: 5 and 40.

↓
0

THE PANEL LIT

−1 0 +1 rows = positions 0…31 · cols = dims 0…15
↓
8

DATA OUT out ↓

Proven: booting…

64 positions → 64 unique encodings, all in [−1,1]; PE(pos+k)=Rk·PE(pos) for every pos, to <1e−9.

Red Team · attacks & breaks
1

THE ADVERSARY WALL

"Why sinusoids? Just append the integer index, or a one-hot position." — The wall: an unbounded index blows up the residual stream and never generalises past the longest position seen in training; a one-hot needs a fixed max length and gives no notion of relative distance. Sinusoids are bounded and admit a position-independent shift operator.
2

THE GRAVEYARD

  • "Absolute position is all you get."
    → shift-by-k is a fixed rotation Rk, so relative offsets are linearly recoverable.
  • "10000 is magic / required."
    → it is an AMBER design constant setting the longest wavelength; other bases work, this one was chosen, not derived.
  • "Two far-apart positions look the same."
    → every one of the 64 rows is distinct (checked); collisions would fail selfcheck.
6

THE TAMPER

Disclosed planted void: swap the sin/cos engine for a raw linear index PE = pos. Encodings go unbounded and the rotation relation collapses — window 7 catches it live.

state: clean