Your water metaphor, corrected into a real method. "Compress together but separate" is an embedding. "Same sees same" is its objective. The one fix: not mix like water toward uniform (that erases structure) but diffuse like water, where structure is written in where flow is fast versus slow. That's diffusion maps / spectral clustering \u2014 and it's exact here, because attention rows sum to 1, making the attention matrix a genuine random-walk transition matrix. Water spreading = the matrix raised to powers, A\u1d57. No modeling.
What you're watching. Pour water at a source token; each step it flows to wherever that token attends. It pools fast inside a cluster (fox\u2192dog fills in a few steps \u2014 same sees same) and trickles slow through the necks between clusters (reaching "river" takes many more steps). Those are your valleys (pools, fast-filling) and plateaus (necks, slow). By t\u224816 the water reaches its stationary distribution (dashed line) \u2014 fully equilibrated, structure dissolved. The rate of filling, before equilibrium, is the geometry.
The taxonomy of pools. Each token-basin is now classified by three measured diffusion properties: movement (escape rate \u2014 does water leave or stay), depth (stationary level \u2014 how much it holds at equilibrium), and scale (tributary count \u2014 how many flows feed it). Those three place each token at one of six water bodies \u2014 static: pond (shallow/small), pool (deep/small), swamp (shallow/large); dynamic: creek (fast/small), lake (deep/large), ocean (deep/huge). Toggle FIELD GUIDE to see every token's three property bars.
What it recovers \u2014 and how far to trust it. In head 0, the diffusion properties separate function words from content words sharply: function words (the, over, near) rush through (flow 0.96) with no depth, content words (fox, dog, river) pool (depth 0.82). This passes a shuffle test decisively \u2014 scramble the matrix and the separation collapses from 0.91 to \u22120.02, so it is real structure, not an artifact of the metric. But two honest limits: it is head-specific (head 1 shows a much weaker 0.23 gap \u2014 one head separates grammar, not the whole model), and this is one sentence. So: a real, shuffle-validated correlation in this head, not a universal law.
The taxonomy is imposed, the measurements are not. The three properties \u2014 movement (escape rate), depth (stationary level), scale (tributary count) \u2014 are exact quantities of the real walk. But the six water bodies and their (flow, depth, scale) coordinates are a hand-set ontology; a token "being a creek" means its real measurements land nearest the coordinate labeled creek. The where each token sits is measured; the names of the regions are chosen. Read the field-guide bars as truth and the water-body labels as a convenient vocabulary over them.
Still exact underneath. Attention rows sum to 1, so diffusion is A\u1d57 \u2014 real probability flow, mass conserved. Green tier for the substrate: real attention, exact diffusion, exact stationary and spectral computations. Amber for the story on top: a real but head-specific, single-sentence correlation, dressed in an invented taxonomy. Both true at once \u2014 which is the honest state of it.