Backprop multiplies a gradient through every layer — and a product of many small numbers races to zero. With sigmoid activations (derivative ≤ ¼), ten layers scale the gradient by (¼)¹⁰ ≈ 10⁻⁶: the early layers barely learn. This is the wall that kept networks shallow for decades. The fixes — ReLU (derivative 1), careful init, residual connections, LSTMs — are the story of deep learning.
THE TECHNIQUE ∏ small derivatives → 0
The demo multiplies the sigmoid’s ¼ gradient through 10 layers (vanishes) vs ReLU’s 1 (survives): live demo
HISTORY & CREDIT Hochreiter · 1991
“More layers always means more power.” — not if the gradient vanishes; a naive deep net can perform worse than a shallow one until the vanishing is fixed. cited
the product · backprop multiplies each layer’s local gradient. the vanish · sigmoid’s ≤¼ over 10 layers → ~10⁻⁶; the depths stop learning. 1991 · Sepp Hochreiter named it — solved by ReLU, init, residuals, LSTM.
Depth multiplies until the signal disappears. the wall
RECOMMEND FOR I-13 the product, on the compiler
On i-13, (¼)¹⁰ = 9.5e−7 (vanished), while 1¹⁰ = 1 (ReLU survives):
$ i13 run nn_vanishing-gradient.i13
RUN OK · 375 step(s) · call depth 11
sig_grad = 9.5e-7 vanishes = 1
relu_grad = 1 relu_survives = 1
Recommend as a NULL — a product identity (B39). The gradient product is fixed arithmetic; the vanishing is its analytic consequence. NULL — why deep was hard.