How do you take a gradient through a random sample? You cannot differentiate the act of sampling — but you can move the randomness out of the parameters. Rewrite z ~ N(μ,σ²) as z = μ + σ·ε with ε ~ N(0,1): now the noise is a fixed input and the gradient of E[f(z)] flows straight through μ and σ. The blue team builds and proves it; the red team tries to break it.
source Kingma & Welling, Auto-Encoding Variational Bayes (2013) — arxiv.org/abs/1312.6114; independently Rezende, Mohamed & Wierstra, Stochastic Backpropagation (2014), arxiv.org/abs/1401.4082. Rendered, not quoted.
Two ways to estimate ∇μ Ez~N(μ,σ²)[f(z)]. Both are unbiased; they differ only in variance.
PATHWISE (reparam): z = μ+σε, so ∂f/∂μ = f′(z). The gradient rides through the sample. SCORE (REINFORCE): f(z)·∂μlog p(z) = f(z)(z−μ)/σ². It only sees f as a black-box weight — and pays for it in variance.
Live, for the current μ,σ (f(z)=zk), over the same fixed-seed draws:
| ∇μ estimate | mean | variance |
|---|
This one identity is what makes the VAE trainable: the encoder outputs μ,σ, the sample z=μ+σε feeds the decoder, and the reconstruction loss backpropagates through the latent draw into the encoder.
It is the low-variance twin of the policy gradient's score-function estimator — same problem (differentiate an expectation over a sampled variable), different geometry. Reparam needs a differentiable, reparameterizable q; score-function works when reparam cannot. Each sphere is the next one's premise.
The blue team's live check: redraw a fixed-seed batch and confirm z = μ+σε still has variance σ² and that reparam variance stays below score variance. If red tampers with ε, this badge is where it shows.
You want to minimize an expectation over a random variable whose distribution you control:
The obstacle: to estimate L you must sample z — and z = sample(μ,σ) has no derivative. The distribution parameters go in; the trouble is getting a gradient back out through the random node.
ε drawn from N(0,1) — the honest noise. Untampered.
Every number is computed on the spot from fixed-seed draws and the closed-form moments — never looked up.
What the machine proves: the pathwise estimator ∇μL ≈ f′(z) is unbiased — its mean equals the true gradient — and for smooth f it has strictly lower variance than the score-function estimator on the same draws. For f(z)=z²: Varrep=4σ², Varscore=μ⁴/σ²+14μ²+15σ² — provably larger by μ⁴/σ²+14μ²+11σ².
The blue team's witness (left) confirms this live; the red team (right) tries to make it false.
It also needs f differentiable in z. Where f is a black box, a reward from an environment, or z is discrete, the high-variance score-function estimator is the only game — which is exactly why policy-gradient RL lives there. "Lower variance" is a property of a regime, not a law.
"Reparameterization always beats REINFORCE." Cut. Only for smooth f and reparameterizable q. For non-smooth or very high-dimensional f the pathwise gradient can have higher variance; there is no free lunch.
"You can reparameterize any distribution." Cut. Only location-scale families and those with a differentiable inverse-CDF. Discrete distributions need a biased relaxation.
"Kingma & Welling invented the pathwise gradient." Kept, corrected. AMBER The pathwise / infinitesimal-perturbation-analysis estimator predates them (Glasserman, Rubinstein). Their contribution was wiring it into deep latent-variable models as the VAE.
The red team's move: draw ε with the wrong variance (2 instead of 1). Now z = μ+σε has variance 2σ², not σ² — the sample no longer matches N(μ,σ²). The witness (window 7) is watching.
Corrupt the noise variance and the reparameterized sample silently mis-represents the distribution; the witness recomputes Var(z), sees it is no longer σ², and turns red. Nothing is faked — the attack is real and it is caught.