To stop a network memorizing its training set, randomly switch off half its neurons on every step. Each pass trains a different thinned sub-network, and the whole ensemble averages at test time — a cheap, powerful regularizer. The trick that keeps the math consistent: inverted dropout scales the survivors by 1/(1−p) during training, so the expected output is unchanged and test time needs no adjustment.
THE TECHNIQUE drop w.p. p; scale survivors by 1/(1−p)
The demo shows inverted dropout with p=0.5 preserves the expected output (i13 grounds the expectation; the coin flip is the wall): live demo
HISTORY & CREDIT Srivastava, Hinton et al. · 2014
“Dropout is used at test time too.” — it is off at inference; the full network runs, which is why the training-time scaling must keep the expectation right. cited
thin the net · zero each neuron with probability p — a different sub-network each pass. keep the expectation · scale survivors by 1/(1−p) so E[output] is unchanged; test time is clean. 2014 · Srivastava, Hinton et al. — regularization by ensembling.
An ensemble of thinned nets, for free. regularization
RECOMMEND FOR I-13 the expectation, on the compiler
On i-13, with p=0.5 a kept neuron scaled by 2, half the time, has expected value 10 = the input:
$ i13 run n2_dropout.i13
RUN OK · 26 step(s)
expected = 10 = x -- E preserved (inverted dropout)
train_test_match = 1
Recommend as a NULL — entropy + a pinned expectation (B47/B39). The mask is random (non-reproducible, B47); the 1/(1−p) scaling that fixes the expectation is pinned. NULL — train a crowd, not a soloist.