Attention has to turn raw similarity SCORES into a set of weights that add up to 1 — how much to look at each token. The softmax does it: exponentiate every score and normalise. A hidden dial, the TEMPERATURE, decides the mood: cold and it snaps to the single best (a hard choice); hot and it spreads evenly (pure indecision). It’s the same function that turns a network’s logits into probabilities. Slide the temperature from decisive to diffuse.
Softmax maps a vector of scores z to a probability distribution: wᵢ = e^(zᵢ/T) / Σⱼ e^(zⱼ/T). The weights are positive and sum to exactly 1 — a valid attention distribution or class probability. The TEMPERATURE T reshapes it: as T→0 it becomes a hard argmax (all weight on the largest score); as T→∞ it becomes uniform (1/n each). It is smooth and differentiable (so gradients flow), and its exponential means a small score gap becomes a large weight gap. A fail-loud self-check throws unless the weights sum to 1, low temperature concentrates on the max, and high temperature flattens toward uniform. ◆ real ML mechanism, node-verified.
Standard temperature-scaled softmax (the exact normalisation and T-limits); attention uses it per query row after the QK scores — the sums-to-1 and temperature-sharpening behaviours are exact.