Attention is a distribution. For each query token, the mechanism softmaxes over all visible keys — the weights sum to 1, a probability distribution over "where this token looks." And it has a head and a tail, just like Zipf, but it's a different curve about a different thing: not how often words appear, but where a token attends. Drag the query slider and watch it shift.
Three features, all documented. The orange spike is an attention sink — token 0 hoards mass (often 30–70% here) regardless of distance, because a content-free anchor is somewhere convenient to dump probability the model doesn't want to spend. It persists across every query: that's the sink ridge in 3D. The amber bar is self/recency — tokens attend strongly to themselves and near neighbors, the ridge running down the causal diagonal. Everything between is the tail: a long stretch of near-zero weights on tokens the query mostly ignores. The diagnostics quantify it — effective support of just a few tokens out of dozens, entropy far below the uniform maximum. Attention is far more concentrated than "attend to everything" suggests.
The hand-off from Zipf. There's no single "attention tail" as a named object, but your instinct caught a real link. Because word frequency is Zipfian, the rare tail words are the ones the model has seen least, so their key/query vectors are the noisiest — and attention over them is least reliable. The Zipf tail of the corpus becomes the undertrained tail of the embeddings becomes the shakiest region of attention. Same word, three curves, one hand-off: frequency poisons representation poisons attention, precisely at the rare end. (Amber tier: these weights are modeled to reproduce published attention structure, not extracted from a specific trained network — the phenomena are real, the exact numbers are illustrative.)