ATTENTION SINK

A pressure-release valve. The head dumps large attention mass onto the first token (often BOS) — not reading content, just parking probability somewhere harmless to keep the softmax stable. Real BERT attention, head 0.

GREEN

2D · Quantitative Channel

CANVAS 2D

demo: sine — replace in draw2D()

CANVAS 2D UNAVAILABLE

3D · Spatial Channel

SOFT-GL · NO GPU · NO LIB

DRAG ROTATE · WHEEL ZOOM

demo: lattice + orbit — replace in build3D()/draw3D()

RENDER FAILED —

Notes

Attention sink. Discovered by Xiao et al. (2023): models route a large fraction of attention onto the first token regardless of content. It is not attending to anything meaningful — it is a stabilizer, a place to dump probability mass when a head has nothing to say, keeping the softmax well-behaved. Here it is measured on real BERT head 0: ~27% of all attention in the sentence lands on token 0. Green tier: real extracted attention.