THE BATCH NORMALIZATION re-centre every layer as you go
As a net trains, each layer’s input distribution keeps shifting under it — a moving target. Batch normalization steadies it: for each mini-batch, subtract the mean and divide by the standard deviation, (x − μ)/σ, so every layer sees inputs with mean 0 and variance 1. It lets you train faster, with higher learning rates and less fuss over initialization — one of the quiet workhorses of modern nets.
THE TECHNIQUE x̂ = (x − μ) / σ
The demo normalizes a batch [2,4,6,8] — the result has mean 0, variance 1 (i13 sqrt via Newton): live demo
HISTORY & CREDIT Ioffe & Szegedy · 2015
“Batch norm works by reducing internal covariate shift.” — that was the original story; later work argues it mainly smooths the loss landscape. cited
the drift · each layer’s input distribution shifts as training changes the layers below. the re-centre · (x−μ)/σ per batch — mean 0, variance 1, a steady target. 2015 · Ioffe & Szegedy — faster, higher learning rates.
A moving target, held still. normalization
RECOMMEND FOR I-13 mean zero, on the compiler
On i-13, the batch [2,4,6,8] has mean 5, var 5, and after (x−μ)/σ the mean is exactly 0:
$ i13 run nn_batch-normalization.i13
RUN OK · 866 step(s) · call depth 42
mean = 5 var = 5 std = 2.236
norm(2) = -1.342 new_mean = 0 mean_zero = 1
Recommend as a NULL — a normalization (B39). (x−μ)/σ is a pinned transform of the batch; every correct implementation agrees. NULL — re-centre every layer as you go.