Cripple the net at random and it learns to stand anyway. During training each unit is deleted with probability p — a different thinned network every step. The trick that keeps it honest is one scalar: survivors are scaled by 1/(1−p), so the expected activation is unchanged and test time needs no correction. That claim is runnable: sample thousands of masks and the mean output lands on the original signal. Terms go in, masks fire, the unbiased signal comes out. The blue team builds it; the red team strips the scaling and watches it drift.
source Srivastava, Hinton, Krizhevsky, Sutskever, Salakhutdinov, Dropout: A Simple Way to Prevent Neural Networks from Overfitting, JMLR 15 (2014) 1929–1958 — jmlr.org/papers/v15/srivastava14a.html (open access; the 2012 co-adaptation report is arxiv.org/abs/1207.0580). Rendered, not quoted.
A layer holds activations x. One training step draws a Bernoulli mask: keep each unit with prob 1−p, zero it with prob p. Inverted dropout then multiplies the survivors by 1/(1−p).
Per unit the expectation is E[out] = (1−p)·(x/(1−p)) + p·0 = x. Unbiased by construction — which is exactly why test time can run the full net untouched.
Live, at the panel's current rate — original vs empirical mean over many masks:
| unit | x | E[out] | bias |
|---|
Dropout is an ensemble of subnetworks sharing weights: a net of n units contains 2ⁿ thinned models, and one forward pass samples one of them. Training averages them; the 1/(1−p) scale at test approximates the ensemble's geometric mean with a single pass.
It sits downstream of backprop — noise injected on the forward pass, gradients still flow the ordinary way through the survivors. Its parent in the arc is the 2012 co-adaptation report: stop feature detectors from co-adapting. Each sphere is the next one's premise.
The blue team's live check: re-sample inverted dropout over many masks at p=0.5 and confirm the mean output still equals the original x (max bias near zero). If red strips the scaling, this badge is where it shows.
The input is a fixed layer of 6 activations and a dropout rate p. Nothing else — dropout needs no learned parameters of its own.
| unit | u0 | u1 | u2 | u3 | u4 | u5 |
|---|---|---|---|---|---|---|
| x | 4 | 3 | 2 | 1 | −2 | −3 |
Each mask is a fresh coin per unit (fixed-seed PRNG, so every count here is reproducible). At rate p, on average a p fraction of these units is zeroed each pass. That layer and that rate are what you feed the panel below.
Inverted: survivors ×1/(1−p). Expected activation preserved — unbiased.
Every number is sampled from real masks on the spot — the mean output is counted, never assumed. Change p or the scaling and the bias recomputes.
What the machine proves, by sampling: with the 1/(1−p) scale the empirical mean output equals the original activation — bias ≈ 0 at every rate. Strip the scale and the expected output is exactly (1−p)·x, biased low; each mask zeros a p fraction of units.
The blue team's witness (left) confirms unbiasedness live; the red team (right) removes the scale to break it.
And the win is not free: convergence slows (paper reports up to ~2–3× more epochs), you need a wider net, and the "average of 2ⁿ models" is an approximation via weight scaling, not an exact ensemble. Dropout is a regularizer, not a proof of anything about the data.
"Dropout averages 2ⁿ networks exactly." Cut. The single-pass weight-scaling inference is an approximation to the ensemble's (geometric) mean — exact only for a single linear layer.
"Dropout is applied at test time." Cut. Off at test — the full net runs deterministically. (MC-Dropout keeps it on deliberately, to sample uncertainty — a different tool.)
"The 1/(1−p) scale is optional bookkeeping." Kept, corrected. Without it the expected activation is (1−p)·x — biased low; the whole layer's signal shrinks. It is the load-bearing line, as window 6 shows.
The red team's move: delete the 1/(1−p) survivor scaling and let inverted dropout run as plain masking. The blue team's witness (window 7) is watching.
Remove the scale and the expected activation collapses to (1−p)·x — at p=0.5 the whole layer is halved. The witness resamples, sees the mean drift off x, and turns red. Nothing is faked; the bias is real and it is caught.