THE WEIGHT INITIALIZATION start so the signal survives
Before a net learns anything, its random weights must be scaled just right, or the signal explodes or vanishes as it passes through layers. The variance of a layer’s output is nin·Var(W)·Var(x), so pick Var(W) = 2/nin (He, for ReLU) and the signal variance is preserved layer after layer. Get this wrong and a hundred-layer net is dead on arrival; get it right and it trains.
THE TECHNIQUE Var(W) = 2/nin (He)
The demo shows a naive init (Var=1) exploding the variance 100×, while He init holds it stable: live demo
HISTORY & CREDIT Glorot 2010 · He 2015
“Just initialize weights small and it’s fine.” — too small vanishes the signal as surely as too large explodes it; the scale must match the fan-in. cited
the variance law · Var(out) = nin·Var(W)·Var(in). the fix · Var(W) = 2/nin (He, for ReLU) or 1/nin (Xavier) — signal preserved. 2010/15 · Glorot & Bengio (Xavier), He et al. — deep nets that train.
The starting point decides whether depth lives. initialization
RECOMMEND FOR I-13 the variance, on the compiler
On i-13, a naive Var(W)=1 over 100 inputs gives Var(out)=100 (explodes); He init gives 2 (stable):
$ i13 run nn_weight-initialization.i13
RUN OK · 28 step(s)
var_out_bad = 100 -- naive init explodes
var_out_he = 2 -- He init 2/n_in, stable
bad_explodes = 1 he_stable = 1
Recommend as a NULL — a variance identity (B39). The signal-variance propagation is fixed algebra; the He/Xavier scale follows. NULL — start so the signal survives.