Expected error splits in three: bias — what the model is too simple to fit; variance — how much the fit wobbles as the training set is redrawn; and noise — the floor no model beats. Push complexity up and bias falls while variance rises, so total error traces a U with a sweet spot in the middle. Down the center the terms go in, the engine resamples and decomposes, the U comes out. The blue team builds and proves it; the red team tries to break it.
source Geman, Bienenstock & Doursat, Neural Networks and the Bias/Variance Dilemma (1992), Neural Computation 4(1), 1–58 — direct.mit.edu/neco/article/4/1/1/5624. Rendered, not quoted.
For squared loss the expected test error at a point splits exactly into three non-negative parts:
E[(y − ĥ(x))²] = bias(x)² + variance(x) + σ²
bias(x) = (mean fit − truth): systematic miss. variance(x) = spread of the fit as the training set is redrawn. σ² = irreducible label noise. Fixed design: 12 training x-locations are pinned; only the noise redraws across sets, so each term is cleanly attributable.
For the model in the panel, the three terms this instant:
| term | value | share |
|---|
This U sits beneath the-double-descent: every model trades fitting the signal against fitting the noise, and the minimum is the truce. It is the same tension that the-vc-dimension counts as capacity and that regularization exists to pull back.
Bias is the approximation side of the capacity story; variance is the estimation side. VC dimension bounds the second; the U shows what it costs the first. Each sphere is the next one's premise.
The blue team's live check: independently re-measure the true expected error by Monte-Carlo with fresh test noise, across all 8 complexities, and confirm it equals the displayed bias² + variance + σ². If red drops the variance term, the decomposition stops balancing and this badge turns red.
Four things feed the engine. Target: f(x) = sin(1.5x + 0.6) on [−2, 2], the signal to be recovered. Noise: labels are y = f(x) + ε, ε ~ Normal(0, σ²), σ = 0.30, so σ² = 0.09. Model class: least-squares polynomials of degree 0…7 — the complexity dial. Protocol: draw 1600 fixed-seed training sets, refit, and read how the prediction shifts.
Bias needs the average fit; variance needs the spread of fits — both demand many training sets, not one. That resampling is the whole apparatus, and it is what you feed the panel below.
Estimated: each term is the Monte-Carlo average over 1600 fixed-seed training sets. Closed-form: the exact fixed-design values. They agree to < 0.01 — that agreement is the proof.
Change the complexity — every term is recomputed by refitting on the spot, never looked up.
What the machine produces, proven: total error bias² + variance + σ² for every complexity 0…7 — bias falling, variance rising, and the total bottoming out at an interior minimum. The irreducible σ² = 0.09 is the floor drawn across.
| deg | bias² | var | σ² | total | error bar (bias² · var · σ²) |
|---|
The blue team's witness (left) confirms these totals live; the red team (right) tries to make them lie.
And the clean three-way split is tied to squared loss. Under 0-1 loss the additive decomposition fails; Domingos (2000) gives a unified form but the terms combine multiplicatively and depend on the loss. Bias and variance are defined against a fixed loss and a fixed data distribution — change either and the numbers move.
"The best model has zero bias." Cut. The highest-degree fit here drives bias² to ~0 but its total (~0.143) is worse than the U-minimum (~0.122). Some bias is optimal — the whole point of the trade.
"Variance is just the noise in the data." Cut. σ² is the label noise; variance is the sensitivity of the fit to which training set was drawn. Two distinct terms — a model can have huge variance on clean-ish data.
"Bias and variance trade off one-for-one." Kept, corrected. They trade in this regime. Regularization, bagging, and overparameterized nets cut variance without paying it all back in bias.
The red team's move: estimate variance from a single training set. One set has no spread, so the variance term reads 0 — and total collapses to bias² + σ², under-counting the true expected error. The blue witness (window 7) is watching.
Drop to a single training set and the wobble becomes invisible: variance → 0, the displayed total sinks below the real error, the witness recomputes and turns red. Nothing is faked; the omission is real and it is caught.