The simplest possible nonlinearity: ReLU(x) = max(0, x) — pass positives, zero negatives. Its derivative is a clean 1 wherever it is active, so gradients flow undiminished through many layers instead of shrinking to nothing (as the old sigmoids did). This one bend is most of why deep networks became trainable, and it is the default activation in nearly every modern net.
THE TECHNIQUE ReLU(x) = max(0, x)
The demo shows ReLU and its derivative — exactly 1 where active, so the gradient does not vanish: live demo
HISTORY & CREDIT Fukushima 1969 · Nair-Hinton 2010
“ReLU neurons never die.” — a neuron stuck in the negative region has zero gradient forever (a “dead ReLU”); leaky/GELU variants soften this. cited
the bend · keep positives, clamp negatives to zero. the gradient · exactly 1 when active — no shrinking, so depth trains (unlike sigmoid’s ≤0.25). 2010 · Nair & Hinton made it standard (roots in Fukushima 1969).
A single kink that made networks deep. activation
RECOMMEND FOR I-13 the gradient, on the compiler
On i-13, ReLU(−2)=0, ReLU(3)=3, and the derivative is 1 where active (the gradient survives):
$ i13 run nn_relu.i13
RUN OK · 52 step(s)
relu(-2)=0 relu(3)=3
grad(3)=1 -- no vanishing
grad(-2)=0 no_vanish = 1
Recommend as a NULL — a pinned function (B39). max(0,x) and its subgradient are fixed; every correct implementation agrees. NULL — the bend that unlocked depth.