Factor the joint into a chain of next-step predictions and the likelihood becomes exact. Every conditional is a classifier; the product of them is the whole distribution. No bound, no implicit density — the number it reports is the true p(x).
source Larochelle & Murray, The Neural Autoregressive Distribution Estimator (AISTATS 2011); journal ext. arxiv.org/abs/1605.02226 · van den Oord et al., Pixel RNN (2016) arxiv.org/abs/1601.06759 Rendered, not quoted.
Pick any ordering of the variables. The chain rule of probability is an identity, exact for every distribution:
Each factor p(xₓ | xₖₖ) is a small predictor over the next variable given everything before it. String them together and you have a generator whose likelihood is computed by multiplication — no marginalisation, no partition function to estimate.
Here the ground truth is a constructed 3-bit joint (a first-order Markov table). The model reads its conditionals straight off that table, so the factorisation is checkable to machine precision.
the-causal-mask is what makes this trainable at scale: mask the attention so token i can only see tokens < i, and a transformer computes every conditional p(xₓ | xₖₖ) in one parallel pass.
The chain rule is the language-model training objective — next-token prediction is exactly this product, and its log is the cross-entropy loss. This sphere is the density backbone that the causal mask industrialised.
Live re-check: recompute the chain-rule product for all 8 outcomes against the joint table, right now, in the current state.
Green only while ∏ p(xₓ|xₖₖ) = p(x) exactly. If window 6 drops the conditioning, this flips red on the next tick.
A constructed, correlated joint over three bits (x₁,x₂,x₃), built as a Markov chain with known transitions. It is not a product of marginals — KL(joint ‖ independent) > 0.
| x₁x₂x₃ | p(x) |
|---|
Factorise under a chosen ordering, then sample one variable at a time (ancestral sampling) with a fixed-seed PRNG.
| x₁x₂x₃ | ∏ conditionals | p(x) |
|---|
Exact likelihood is not free. The chain rule is exact only for the conditionals you actually have. In the real setting each p(xₓ|xₖₖ) is a learned approximation, and errors compound multiplicatively along the sequence.
Sampling is inherently sequential — O(n) passes, one variable at a time. PixelRNN pays this cost per pixel; it is why autoregressive generation is slow next to a one-shot flow or a parallel decoder.
The ordering is a modelling choice: every ordering is a valid factorisation of the same joint, but with imperfect conditionals different orderings give different approximations.
“A generative model must marginalise over a latent to score a point.”
→ Autoregressive models have a tractable exact likelihood; the score is a product of conditionals, no latent integral.
“You need the ‘right’ variable ordering.”
→ Any ordering factorises the joint exactly (window 0 verifies two orderings reconstruct it). Ordering only matters once conditionals are approximate.
“Conditioning is just an optional refinement.”
→ Drop it and you assert independence. For a correlated joint the product then no longer equals p(x) — see window 6.
Planted void (disclosed): replace each conditional p(xₓ|xₖₖ) with the bare marginal p(xₓ) — i.e. assert the bits are independent.
For this correlated joint the product of marginals is not the joint, so window 0's reconstruction breaks and the Witness (7) catches it live.