The test-error curve that dips, spikes at the interpolation threshold p = n, then dips again. Random-feature linear regression, fit in closed form: ordinary least squares while p < n, the minimum-norm interpolant once p ≥ n. Rendered, not quoted.
source Belkin, Hsu, Ma & Mandal — Reconciling modern machine-learning practice and the bias-variance trade-off (2019) · arxiv.org/abs/1812.11118
Draw n = 15 training points from a smooth target f(x) = sin(1.7x) + 0.35x on [-3, 3], plus fixed-seed noise (σ = 0.1). Build random Fourier features φj(x) = √2 cos(wjx + bj), wj ~ N(0, 3²).
Fit a linear model on the first p features. For p < n: ordinary least squares (β = (ΦᵀΦ)⁻¹Φᵀy). For p ≥ n: the minimum-norm interpolant β = Φᵀ(ΦΦᵀ)⁻¹y — closed form, deterministic. No stochastic training.
Extends the-bias-variance-tradeoff past the point where classical theory stops. The old U-curve says: more parameters → lower bias, higher variance → overfitting. It ends at p = n.
This sphere continues the axis beyond interpolation. Past p = n the minimum-norm solution smooths out, variance falls again, and over-parameterized models — which the trade-off predicts should overfit — generalize.
Re-runs the shape test live on the reported curve: is the test error at p = n a strict local peak above a classical p and above the over-parameterized tail? Confirms LIT; flips red the instant the panel is tampered.
15 noisy training points · a sweep of feature counts p = 1 … 140 · the interpolation threshold p = n = 15 marked.
…
Train error → 0 at interpolation, yet test error peaks there and then descends a second time. The proven shape.
wall
"More parameters than data points must overfit." True for a generic interpolant — pick a bad one and the peak never ends. The result is fragile: it holds only for the minimum-norm solution and a benign feature spectrum. Swap in features that are too collinear near p = n and the solve loses all precision; too high-frequency and the second descent vanishes. Double descent is real but conditional.
"Bigger models always overfit."
→ past interpolation, the min-norm interpolant's risk falls again.
"The peak is a training bug."
→ training error is exactly ~0 there; the peak is in test error, driven by the min-norm weight-norm blowing up at p = n.
"Any interpolant shows this."
→ only min-norm. A non-min-norm interpolant fits the same data with a larger ‖β‖ and worse test error (window 6 · verified).
Disclosed planted void: report the training error as if it were the test error. Training error is monotone to 0, so the p = n peak disappears — the double-descent shape is faked flat.
The Witness (7) recomputes the shape and catches it live.