An autoencoder that learns a distribution, not a point. An encoder maps each datum to a Gaussian q(z|x) = N(μ, σ²); a decoder maps z back to x; both are trained end-to-end by one scalar — the negative ELBO = reconstruction + KL(q‖N(0,1)). For a linear-Gaussian model every quantity is closed-form, so the machine computes exact numbers, never trained guesses. Down the center: a datum goes in, the ELBO is evaluated, the proven bound comes out. The blue team builds it; the red team breaks it.
source Kingma & Welling, Auto-Encoding Variational Bayes (2013) — arxiv.org/abs/1312.6114. Rendered, not quoted.
A prior p(z)=N(0,1), an encoder q(z|x)=N(μ,σ²), a decoder p(x|z)=N(g(z),1). Take the decoder linear — g(z)=W·z+B with fixed W=1.4, B=0.5 — and the whole model becomes exactly solvable.
Then the KL to the prior is closed-form:
KL(N(μ,σ²)‖N(0,1)) = ½(σ² + μ² − 1 − log σ²)
The reconstruction is the expected decoder NLL, also closed-form because z is Gaussian; and the marginal p(x)=N(B, W²+1) and true posterior p(z|x) are Gaussians too — so nothing here is estimated.
| piece | form | value now |
|---|
the-elbo was a scalar — a lower bound on log p(x). Kingma & Welling made it an architecture: an encoder network emits μ,σ; a decoder network reads z; and the the-reparameterization-trick writes z = μ + σ·ε, ε∼N(0,1), so a gradient can flow through the sampling.
That one move — sample outside the graph, scale inside it — turned the ELBO into the first scalable deep generative model. Each sphere is the next one's premise.
The blue team's live check: re-evaluate the bound over a grid of data and encoders and confirm the invariants — KL(0,1)=0, −ELBO ≥ NLL everywhere, equality only at q=posterior. If red flips a sign, this badge is where it shows.
The input is a single scalar datum x. The decoder is fixed once — p(x|z)=N(W·z+B, 1), W=1.4, B=0.5 — so the only free choice is the encoder you feed in: a mean μ and a spread σ, plus a KL weight β.
The reparameterized reconstruction is exact: Eq[½(x−Wz−B)²] = ½((x−Wμ−B)² + W²σ²). That is what the panel below turns into a loss — no Monte-Carlo noise, no training loop.
The optimal encoder for weight β is closed-form: μ*=W(x−B)/(W²+β), σ*²=β/(W²+β). At β=1 that is the true posterior and the bound is tight.
Every number is computed from the closed forms on the spot — recon, KL, −ELBO, NLL, and the gap KL(q‖posterior). Nothing is looked up or trained.
What the machine proves, for any encoder you dial in: the loss −ELBO = recon + β·KL; at β=1 it equals NLL + KL(q‖posterior), so it upper-bounds the NLL and is equal only at q=posterior. KL(q‖prior) ≥ 0 always, =0 only at N(0,1). Raising β pulls q* toward the prior — the β-VAE behaviour.
The blue team's witness (left) confirms these live; the red team (right) flips the KL sign to break them.
Two more real failures: the amortization gap (one encoder network must fit every x, and it won't hit each true posterior), and the blurry Gaussian decoder — a fixed-variance Gaussian likelihood averages modes, so samples smear. None of these are bugs in this engine; they are the honest cost of the method.
"The VAE maximizes the likelihood." Cut. It maximizes a lower bound; the shortfall is exactly KL(q‖posterior) ≥ 0, zero only when q hits the true posterior — shown live in the gap.
"The KL term is a regularizer you can tune to zero." Kept, corrected. Set β=0 and you get a plain autoencoder with an unstructured latent — the KL is what makes z a usable distribution you can sample.
"Blurry reconstructions mean the inference is broken." Cut. A Gaussian decoder's optimum is the mean; the blur is the likelihood's fault, not the ELBO's. Swap the decoder, not the bound.
The red team's move: flip the KL sign — subtract it instead of adding, rewarding divergence from the prior. The posterior flees N(0,1), KL≥0 stops constraining, and the loss dives below the NLL. The blue team's witness (window 7) is watching.
Flip the sign and −ELBO no longer equals recon+KL and no longer upper-bounds the NLL — the witness recomputes, finds a case where the loss falls under the NLL, and turns red. Nothing is faked; the attack is real and it is caught.