THE SOFTMAX — a dart

THE SOFTMAX a vector becomes a choice

To turn a network’s raw scores into a probability distribution over classes, exponentiate and normalize: softmax(x)ᵢ = eˣᵢ / Σ eˣʲ. Every output lands in (0,1) and they sum to 1, with the largest score getting the lion’s share — a soft, differentiable version of “pick the max.” It is the final layer of nearly every classifier, and the same Boltzmann form that statistical physics uses for states.

THE TECHNIQUE softmax(x)ᵢ = eˣᵢ / Σ eˣʲ

The demo softmaxes [1,2,3] — the outputs sum to 1, the biggest score wins: live demo

HISTORY & CREDIT Boltzmann / Gibbs · Bridle 1990

“Softmax picks the maximum.” — it is a soft argmax: every class keeps some probability, which is what makes it differentiable and trainable. cited

exponentiate · eˣ makes everything positive and amplifies gaps.
normalize · divide by the sum — a distribution that sums to 1.
Boltzmann · the same form as thermodynamic state probabilities; named in NNs by Bridle (1990).

Scores, turned into a distribution. activation

RECOMMEND FOR I-13 the distribution, on the compiler

On i-13, softmax([1,2,3]) = [0.090, 0.245, 0.665] — sums to 1, monotone, largest score dominant:

$ i13 run nn_softmax.i13 RUN OK · 1948 step(s) · call depth 31 p = [0.0900, 0.2447, 0.6652] total = 1 monotonic = 1 p3_biggest = 1

Recommend as a NULL — a pinned map (B39). softmax is a fixed function of the logits; every correct implementation returns the same distribution. NULL — a vector becomes a choice.