How a neural net learns which way to nudge each weight: a forward pass computes the output and its error, then a backward pass applies the calculus chain rule from the loss back to every weight — reusing shared results so all the gradients cost about one forward pass. That is the whole trick. It is reverse-mode automatic differentiation, and it is older and less Hinton’s than the story says.
THE TECHNIQUE forward compute, backward differentiate
A single sigmoid neuron: z = w·x + b, a = σ(z), loss L = ½(a − t)². Forward gives a and L; backward gives the gradient by the chain rule dL/dw = (a−t)·a(1−a)·x. Drag the weight; watch the loss and its gradient, then take gradient-descent steps down the curve. live demo
HISTORY & CREDIT 1986 made it famous; 1970 invented it
“Hinton invented backprop in 1986” — the 1986 paper invented popularised it; the algorithm is 16 years older, and the name is older still. cited
1676 / 1847 · Leibniz gives the chain rule; Cauchy gives gradient descent — backprop adds no new calculus and is not the learning step (that is descent), only the gradient. 1960s · Kelley & Bryson compute chained gradients in optimal control (the adjoint method) — the idea is born in aerospace, not AI. Rosenblatt even coins “back-propagating error” (1962) — for a different scheme. 1970 · Seppo Linnainmaa (MSc thesis) writes down the modern reverse mode of automatic differentiation, with code — but for rounding error, no neural nets. The engine, invented. 1974 / 1986 · Paul Werbos first proposes applying it to neural nets (the explicit NN application follows in 1982); Rumelhart, Hinton & Williams (Nature) make it famous and show it learns hidden representations — a rediscovery, not the first.
Every autograd today — PyTorch, TensorFlow — is Linnainmaa’s 1970 reverse mode at scale. Linnainmaa 1970 / Werbos 1974
RECOMMEND FOR I-13 multiply, add, and the chain rule
Forward and backward are both f64 multiply/add — and the single-neuron gradient comes out exactly on the compiler:
$ i13 run bp.i13 # x=1, w=0.5, b=0, t=0
a = 0.62245933 L = 0.19372781 dL/dw = 0.14628025 # sigmoid via std/exp
Recommend: the arithmetic runs — weighted sums forward, a product of local derivatives backward (verified dL/dw = 0.14628). The one transcendental is the nonlinearity: sigmoid needs exp (the stdlib Taylor std/exp.i13 supplies it), or swap in ReLU (a max, no exp). The weight matrices of a real net are 2-D — PS-004 — flattened to 1-D f64 arrays. Note: backprop only supplies the gradient; the learning is gradient descent (Cauchy) stepping the weights — two pieces the story routinely fuses into one.