Is a head staring at ONE token, or glancing at everything? Measure the ENTROPY of its attention weights: low entropy means a sharp, decisive focus; high entropy means the gaze is smeared across many tokens. It’s a single diagnostic number that separates the precise ‘pointer’ heads from the diffuse ‘averaging’ ones — and flags a head that has collapsed onto a sink. Slide from a laser focus to a wide blur.
The entropy of an attention distribution H = −Σᵢ wᵢ log wᵢ quantifies how spread-out the gaze is: it is 0 when all weight is on one token (a perfectly sharp pointer) and maximal, log n, when the weight is uniform over all n tokens (a pure average). e^H is the ‘effective number’ of tokens attended. This one scalar cleanly distinguishes sharp retrieval/pointer heads (low H) from diffuse mixing heads (high H) and is used to monitor whether a head has collapsed onto an [[attention-sink]]. A fail-loud self-check throws unless a peaked distribution has strictly lower entropy than a uniform one, and the uniform entropy equals log n. ◆ real ML diagnostic, node-verified.
The Shannon entropy of the weight row (the exact 0-to-log-n range and effective-span e^H); interpreting HIGH vs LOW as ‘averaging’ vs ‘pointer’ is the standard reading, not a proof of function — the entropy computation is exact.