One knob sits between certainty and noise. Divide the logits by T before the softmax and the whole distribution swings: as T→0 it collapses onto the single best token (argmax); as T→∞ it flattens to the uniform — every token equally likely, entropy at its ceiling log n. The logits go in, the engine reshapes them, the distribution comes out. Blue builds and proves the shape; red tries to bend the curve.
source Ackley, Hinton & Sejnowski, A Learning Algorithm for Boltzmann Machines, Cognitive Science 9 (1985) 147–169 — the Boltzmann/softmax sampling rule doi.org/10.1207/s15516709cog0901_7. AMBER: archive.org holds no scan of this specific paper (searched title + authors, 0 hits); DOI cited instead. Rendered, not quoted.
Given fixed logits z and temperature T > 0, the probability of token i is
pi = ezi/T / Σj ezj/T
Two invariants pin it down: Σp = 1 always, and the ranking of p never changes — T only stretches or compresses the gaps, it never reorders. The Shannon entropy H = −Σ p ln p measures how spread the mass is.
Live, for the current T:
| quantity | value |
|---|
This is not a lone trick. The identical ÷T lives inside the-attention's scaled dot-product — there it is a fixed √d, keeping the score softmax from saturating.
Pulled out and made a live knob, the same operation becomes the sampler's creativity dial: T<1 sharpens toward greedy decoding, T>1 loosens toward exploration. One mechanism, two seats — the attention layer's silent constant is this sphere's exposed lever. Each sphere is the next one's premise.
The blue team's live re-check: recompute the distribution and entropy at the concentration limit, the uniform limit, and along a monotone grid, and confirm each against the values proven at boot. If red bends the curve, this badge catches it.
Five fixed, distinct logits — the model's raw, unnormalized scores for five candidate tokens. Nothing here is random; these are the constants the engine reshapes:
| token | logit zi |
|---|
The largest score belongs to — — the argmax, the token greedy decoding would always pick. Temperature decides how much the other four get a say.
—
entropy — nats of ceiling —
Drag T — every bar is recomputed from ez/T/Σ on the spot, never looked up. The shape you see is the softmax, live.
What the machine produces, proven: a valid probability distribution for every T>0 (Σp = 1), collapsing to a one-hot on the argmax as T→0⁺ and rising to the uniform (entropy = log 5 ≈ 1.6094 nats) as T→∞ — with entropy strictly monotone increasing in T the whole way between.
The blue team's witness (left) confirms these limits live; the red team (right) tries to make the curve lie.
And T is global — one scalar over the whole vocabulary. It cannot ban a specific bad token (that is logit-bias / top-p / top-k) nor rescue a miscalibrated model. High T buys diversity by paying with coherence; the trade is real and unavoidable from this knob alone.
"T = 0 gives a uniform, safe distribution." Cut. Backwards — T→0⁺ is maximally peaked (one-hot on the argmax); T→∞ is uniform. The engine's monotone-H proof settles the direction.
"Higher temperature adds information / more entropy is better." Cut. Entropy rises, but it is spread, not signal — H climbs toward log n precisely as the model's preference is erased.
"Temperature can change the most likely token." Cut. Softmax is order-preserving in z; ÷T is a positive rescale. The argmax is fixed for all T>0 — proven in the witness.
The red team's move: swap the scaling from ÷T to ÷T². Limits still look right, but the whole sharpening curve is wrong — the entropy at a given T no longer matches the truth. The blue witness (window 7) is watching.
Under ÷T², the distribution still sums to 1 and still hits both limits — so it hides at the edges — but at T=0.5 and T=2 the entropy is provably off. The witness recomputes, disagrees with the boot-proven values, and turns red. Nothing is faked; the attack is real and it is caught.