Deep nets stall because every layer’s input distribution keeps drifting as the layers below it learn. The fix is almost embarrassingly plain: before the nonlinearity, standardize the batch — subtract its mean, divide by its spread, then let the network rescale with two learned knobs. Down the center, data flows: raw activations go in, the normalizer standardizes, the shaped output comes out. The blue team builds and defends it; the red team tries to break it.
source S. Ioffe & C. Szegedy, Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift (2015) — arxiv.org/abs/1502.03167. Rendered, not quoted.
Over a mini-batch of activations for one feature, four steps, all differentiable:
1 mean μ = average of the batch. 2 variance σ² = mean squared distance from μ (population, ÷N). 3 normalize: x̂ = (x−μ) / √(σ²+ε) — now mean 0, variance 1. 4 re-shape: y = γ·x̂ + β, where γ and β are learned, so the layer can undo the normalization if that is what helps.
This batch, live (ε = 1e-8 AMBER chosen — the paper uses ~1e-5; smaller ε here buys a cleaner unit-variance demo):
| μ (in) | σ² (in) | mean (out) | var (out) |
|---|---|---|---|
| — | — | — | — |
Normalize the input to every layer and gradients stop vanishing or exploding down a long stack — the drift each layer sees is bounded. That stabilization is exactly what let genuinely deep nets be trained at all.
The next stone in the arc, the-skip-connection (ResNet), stacks that idea past 100 layers: BN keeps each layer’s signal well-conditioned, the skip gives the gradient a clean path home. Each sphere is the next one’s premise.
The blue team’s live check: take the current batch, run it through the normalizer at γ=1, β=0, and confirm the output is mean 0, variance 1. If red tampers, this recomputed number stops equalling 1 — and the badge turns red.
The input is a mini-batch: for one feature (channel), the values it takes across the N examples currently in the batch. Here N = 32, drawn from a fixed-seed generator so the run is reproducible. Their raw mean and spread are whatever the layers below happened to produce — typically not centered, not unit-scale.
That drifting, un-standardized distribution is the problem batch norm is fed to fix.
Move the sliders — the output statistics are recomputed from the batch on the spot, never looked up. At γ=1, β=0 the output is standardized; β then sets the mean, |γ| sets the std.
What the machine produces, proven on every batch: at γ=1, β=0 the output has mean 0 and variance 1 (to 1e-6); turning the knobs moves the mean to exactly β and the std to exactly |γ|. A constant batch (σ²=0) does not blow up — ε guards the divide. The current output is above; these invariants are the result.
The blue team’s witness (left) confirms mean0/var1 live; the red team (right) tries to make it false.
Because it couples examples within a batch, it also leaks batch composition into each prediction — a real dependency the plain feed-forward story hides. BN is a powerful default, not a law.
“BN works by reducing internal covariate shift.” Cut. Santurkar et al. (2018) injected noise after BN — shift restored, training still faster. The measured effect is a smoother loss landscape, not less covariate shift. The paper’s own name is the disputed part.
“BN behaves the same at test time.” Cut. Training uses the batch’s stats; inference uses running averages collected during training. Different math — a classic eval-mode bug.
“Just center and scale — γ, β are cosmetic.” Kept, corrected. Without the learned γ, β the layer can only output mean0/var1; the two knobs let it recover any distribution, including the identity.
The red team’s move: skip the divide by √(σ²+ε) — only subtract the mean. The output is centered but keeps the batch’s raw variance, so it is not unit-scale. The blue team’s witness (window 7) is watching.
Skip the normalization divide and variance no longer equals 1 — the witness recomputes, disagrees with the invariant, and turns red. Nothing is faked; the attack is real and it is caught.