Corrupt the data, then learn to un-corrupt it. Vincent's 2008 move looks like a trick and turns out to be a theorem: the best reconstructor of a noised point is the posterior mean E[x | x̃], and its correction vector r*(x̃) − x̃ is exactly σ² ∇log p(x̃) — it points back up the density, toward where the data lives. Down the center: clean points go in, get noised, the optimal map pulls them out onto the manifold. The blue team proves the map is the score; the red team trains it at zero noise so it learns nothing.
source Vincent, Larochelle, Bengio & Manzagol, Extracting and Composing Robust Features with Denoising Autoencoders, ICML 2008, 1096–1103 — doi:10.1145/1390156.1390294 (no stable arXiv id; DOI cited, marked AMBER). The score identity is Vincent 2011 & Alain–Bengio 2014. Rendered, not quoted.
Data is Gaussian, x ∼ N(μ, σd²); corruption adds independent noise, x̃ = x + ε, ε ∼ N(0, σn²). The reconstructor that minimizes E∥r(x̃)−x∥² is not learned by luck — it is the conditional mean, and for Gaussians it is closed-form:
r*(x̃) = a x̃ + (1−a)μ, a = σd² / (σd²+σn²)
A precision-weighted average of the noisy point and the prior mean. The map is a contraction toward the data (slope a < 1), and its Bayes risk is exactly a σn² per coordinate.
Subtract the input from the reconstruction and something familiar appears:
r*(x̃) − x̃ = σn² ∇log p(x̃)
The correction is the score of the corrupted density, scaled by the noise variance — Vincent (2011), Alain–Bengio (2014). A denoiser is a score estimator. Scale it across many noise levels and you have the-score-matching, which the-diffusion-model ran with. Each sphere is the next one's premise.
The blue team's live check: re-derive the correction field from the current controls and confirm r*−x̃ = σn²∇log p to machine precision at sample points. If red trains at zero noise, the field collapses and this badge turns red.
Two isotropic Gaussians in the plane, mean at the origin. Clean points are drawn from the data density N(0,σd²I); each is then kicked by fresh noise N(0,σn²I) to give a corrupted point off the manifold. Every sample uses a fixed-seed PRNG, so the picture is deterministic and the self-check reproducible.
The task the machine is handed: given only a corrupted point, name the clean one it most likely came from.
Pink arrows are the correction r*(x̃)−x̃ at grid points — every one points inward, up the density, toward the clean cloud. Drop σn to 0 and the arrows vanish: on the manifold the optimal map is the identity.
Every number is computed live from the two σ values — contraction a, Bayes risk aσn², and the max mismatch between the correction and σn²∇log p across the grid. Nothing is looked up.
What the machine produces, proven: the optimal denoiser is the posterior mean (checked against numeric integration to <1e−4), its correction equals the score exactly (<1e−9), it is the identity at zero noise, it strictly beats the do-nothing map in Monte-Carlo MSE, and larger corruption forces a smoother (more contractive) map that keeps only coarse structure.
The blue team's witness (left) re-derives the score identity live; the red team (right) tries to make the field lie.
The score identity is exact only in expectation and cleanest for small σ; a single fixed noise level sees the density through one blur and misses fine structure — the reason diffusion needs a ladder of noise scales, not one denoiser. And a denoiser trained on the wrong noise level is confidently wrong.
"An autoencoder just learns to copy its input." Cut. A plain autoencoder can; a denoising one cannot — the corruption forbids the identity and forces it to model where the data lives.
"The denoiser removes the noise it was given." Corrected. It cannot recover the specific ε; it returns the mean of all clean points that could have produced x̃ — the manifold projection, not the inverse of one draw.
"Bigger corruption is strictly better regularization." Cut. Bigger σn smooths more (a→0, map→constant mean); past a point it erases the very structure you wanted — a bias–variance knob, not a free lunch.
The red team's move: train the denoiser at zero corruption. Then its optimal map is the plain identity — it reconstructs perfectly on clean data and learns nothing about the manifold: r−x̃ = 0, no score, no inward field.
Set the training noise to zero and the arrows collapse to nothing. The witness (window 7) re-derives the score identity, finds r−x̃ = 0 ≠ σn²∇log p, and turns red. The attack is real and it is caught.