Learn the shape of a distribution without ever computing its normalizer. The score is the gradient of the log-density, s(x) = ∇x log p(x), and the constant Z vanishes under that gradient. Hyvärinen's trick: match the model's score to the data's score, using an objective that — by integration by parts — equals the Fisher divergence up to a model-independent constant, and never once mentions Z. Down the center, data flows: the samples go in, the objective is computed, the estimator comes out. The blue team builds and defends it; the red team tries to break it.
source Hyvärinen, Estimation of Non-Normalized Statistical Models by Score Matching (2005), JMLR 6:695–709 — jmlr.org/papers/v6/hyvarinen05a.html (no arXiv; 2005 JMLR). Rendered, not quoted.
For a Gaussian N(μ, σ²) the true score is closed-form:
s(x) = ∇x log p(x) = −(x − μ)/σ²
The model is the linear-score family sθ(x) = a(x − μ); the truth is a* = −1/σ². Hyvärinen's implicit objective in 1-D:
J(a) = Ep[½ sθ(x)² + s′θ(x)]
Live, from the current panel (closed-form ↔ numeric grid):
The move here — keep ∇ log p, drop the partition function Z — is exactly what the-diffusion-model runs at every noise level: a network learns the score of each noised marginal, then Langevin/reverse-SDE walks up it. It is also what made the energy-based world trainable — an energy E(x) defines p ∝ e−E with an intractable Z, but its score ∇x(−E) has no Z at all.
Each sphere is the next one's premise: score matching is the estimator, denoising score matching (Vincent 2011) the bridge, diffusion the payoff.
The blue team's live check: re-minimize the current objective over the score coefficient a and confirm the minimizer lands on a* = −1/σ². If red drops the trace term, the minimizer collapses to a → 0 and this badge turns red.
All the machine ever sees is data drawn from pdata = N(μ, σ²) — never its normalizer. It has:
Set the truth below. μ and σ define pdata; the panel then hunts for a.
Objective: J(a) = ½ a²σ² + a (trace term present). Minimized exactly at a* = −1/σ².
J(a) across the coefficient (min marked):
| a | J(a) | Fisher ½E[(sθ−s*)²]+C |
|---|
Change any control — J is computed on the spot (closed form checked against a numeric Gaussian grid), never looked up. Z appears nowhere.
What the machine produces, proven: the score-matching objective J(a) equals the Fisher divergence ½Ep[(sθ−s*)²] plus a constant C = −½/σ² that does not depend on a — so its minimizer is the true score, a* = −1/σ², recovered without ever touching Z.
The blue witness (left) re-minimizes live; the red team (right) tries to make the minimizer wrong.
The trace/Hessian term costs O(d) extra backprops in d dimensions — prohibitive at scale, which is why sliced (Hutchinson) and denoising score matching exist. It also needs a smooth density on all of Rd: sharp manifolds and boundaries break the integration-by-parts that makes the trick work.
“You still need the normalizing constant Z.” Cut. The score of log p drops log Z (a constant in x); the objective is a sample average of s and s′ only — Z never appears.
“Score matching hands you the density p(x).” Cut. It hands you ∇ log p. Recovering p still needs integrating and normalizing — that is the deliberately skipped step.
“The trace(∇s) term is optional regularization.” Cut. Drop it (window 6) and the identity to Fisher divergence dies; the objective is minimized by the degenerate s → 0, not by the truth.
The red team's move: delete the trace term s′θ(x) from the objective, leaving only ½E[sθ²]. Without it the objective is no longer the Fisher divergence — it is minimized by shrinking the score to zero. The blue witness (window 7) is watching.
Delete s′ and the minimizer slides from a* = −1/σ² to a = 0 (the flat, useless score). The witness re-minimizes, disagrees with the truth, and turns red. Nothing is faked; the attack is real and it is caught.