A joint distribution can be impossible to sample directly, yet its full conditionals — one coordinate given all the rest — are easy. The Gibbs sampler walks the space one axis at a time, redrawing each coordinate from its conditional, and the chain converges on the joint you could not touch head-on. Down the center, data flows: the target correlation goes in, the sampler sweeps, the recovered cloud comes out. The blue team builds and defends it; the red team tries to break it.
source S. Geman & D. Geman, Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images (1984), IEEE Trans. PAMI, vol. PAMI-6, no. 6, pp. 721–741 — doi.org/10.1109/TPAMI.1984.4767596. Rendered, not quoted.
For the standard bivariate normal with correlation ρ, the conditional is closed-form and Normal:
x | y ~ N( ρ·y , 1−ρ2 )
This is not asserted — it is derived from the joint precision matrix Λ = Σ−1: the conditional variance is 1/Λxx and the mean coefficient is −Λxy/Λxx. Live, for the current ρ:
| quantity | from Λ | closed form |
|---|
Draw x from that Normal, then y from its mirror, and repeat.
Gibbs is the Metropolis sampler with the acceptance step erased. Metropolis proposes a move and accepts it with probability min(1, ratio); when the proposal is the exact full conditional, that ratio is always 1 — every move is accepted.
So Gibbs is the special case where each coordinate can be sampled exactly. That is the engine beneath Bayesian hierarchical models and LDA topic modelling: sample each latent given the others, sweep, repeat. Each sphere is the next one's premise.
The blue team's live check: re-run the sampler on a fixed seed and confirm the recovered variance, mean and correlation match the target within tolerance. If red swaps in the wrong conditional variance, this badge is where it shows.
The joint you cannot sample directly here is the standard bivariate normal — two unit-variance Gaussians tied by correlation ρ:
| Σ | x | y |
|---|---|---|
| x | 1 | ρ |
| y | ρ | 1 |
You feed one number: the target correlation ρ. Both marginals are standard Normal; the whole shape is that single off-diagonal. That is what the panel below must reconstruct without ever drawing from the joint.
Each step draws x from N(ρ·y, 1−ρ2), then y from N(ρ·x, 1−ρ2). Fixed-seed PRNG — nothing is looked up, the cloud is genuinely sampled.
| moment | sampled | target |
|---|
Move ρ — the conditionals recompute and the chain re-sweeps on the spot; the recovered moments are measured off the chain, never assumed.
What the machine produces, proven: from full conditionals alone, the chain recovers the target it never sampled directly — marginals standard Normal (mean 0, variance 1), ρ recovered within tolerance, and the target moments a fixed point of a full Gibbs sweep. The current ρ's recovered cloud is above; the invariance is the output.
The blue team's witness (left) confirms these numbers live; the red team (right) tries to make them wrong.
And it needs the conditionals to be samplable in the first place. Without conjugacy you fall back to Metropolis-within-Gibbs, and the free-lunch acceptance is gone. Gibbs is not the sampler; it is the lucky case where every conditional is easy.
"Gibbs samples the joint distribution directly." Cut. It never touches the joint — it only ever draws from one-dimensional conditionals, and the joint emerges as the chain's stationary law.
"Every draw is an independent sample." Cut. Successive draws are correlated (here lag-1 autocorrelation is ρ2); the effective sample size is smaller than the count, which is why the witness uses a tolerance, not equality.
"Convergence is guaranteed to be fast." Kept, corrected. Convergence holds under mild conditions, but the rate can be arbitrarily slow — the adversary's correlated and multi-modal cases are exactly where.
The red team's move: swap the conditional variance from 1−ρ2 to plain 1 — a plausible-looking bug. The sweep no longer leaves the target invariant; the stationary variance blows up to 1/(1−ρ2). The blue team's witness (window 7) is watching.
Use the wrong conditional variance and the recovered variance drifts off 1 while ρ no longer means what it should — the witness recomputes, disagrees with the target, and turns red. Nothing is faked; the attack is real and it is caught.