Apply the Bellman operator until the values stop moving. Start from nothing — V₀ = 0 — and iterate Vk+1 = T Vk on a small Markov decision process. Because T is a γ-contraction, the error to the true optimum V* is crushed by a factor of γ every sweep: the estimates march, geometrically and unstoppably, to the one value function that satisfies the Bellman equation. Down the center, data flows: the model goes in, the operator iterates, the fixed point comes out. The blue team builds and defends it; the red team tries to break it.
source Richard Bellman, Dynamic Programming (Princeton University Press, 1957) — archive.org/details/dynamicprogrammi0000bell. Rendered, not quoted.
Validity is not memorized; it falls out of one map. The optimality operator T acts on a value vector:
(T V)(s) = maxa [ R(s,a) + γ · Σs′ P(s′|s,a) · V(s′) ]
Because that max-of-affine map is non-expansive in the max and 1-Lipschitz in the sum, and the sum is scaled by γ<1, T is a γ-contraction in the sup-norm: ‖TU − TV‖∞ ≤ γ‖U − V‖∞. That single fact is the whole guarantee.
Live Vk for the current sweep (the panel drives this):
| state | Vk | V* |
|---|
The Bellman equation is a fixed-point condition: V* = T V*. This sphere turns that condition into a procedure.
Banach's fixed-point theorem, iterated: a contraction on a complete space has exactly one fixed point, and repeated application from any start converges to it. Value iteration is Banach applied to T — the equation was the premise, the iteration is the proof made runnable. Each sphere is the next one's premise.
The blue team's live check: re-run the whole iteration from V₀ = 0 and confirm the error contracts by γ every step, decreases strictly, and lands on the closed-form V*. If red cancels the discount, this badge is where it shows.
A tiny deterministic Markov decision process: three states A B C, each with two actions (reward, next state). The discount is γ; the start is V₀ = 0.
| state | action | reward | → next |
|---|---|---|---|
| A | stay | 1 | A |
| A | go | 0 | B |
| B | stay | 3 | B |
| B | go | 0 | A |
| C | go | 5 | B |
| C | stay | 0 | C |
At γ = 0.9 the optimum is closed-form: B loops on reward 3 → V*(B)=30; A prefers B → V*(A)=27; C jumps to B → V*(C)=32. Feed this into the operator below.
Untampered: each sweep multiplies the error by at most γ. Banach guarantees convergence from any V₀.
| k | V(A) | V(B) | V(C) | ‖Vk−V*‖∞ | ≤ γk·e₀ |
|---|
Every number is computed live from the operator on the spot, never looked up. Watch the error column fall by the factor γ each sweep.
What the machine produces, proven: from V₀ = 0 the iteration converges to the unique V* satisfying V* = T V*. At γ = 0.9 that is exactly [27, 30, 32], matched to 1e-8; the sup-norm error is bounded above by γk·e₀ at every step k and is strictly decreasing.
The blue team's witness (left) re-derives this live; the red team (right) tries to cancel the discount and break the contraction.
It also converges only geometrically: the rate is exactly γ, so as γ → 1 (the case you most want) it becomes arbitrarily slow, and it reaches V* only in the limit, never in finitely many exact steps. Policy iteration terminates in finitely many iterations; value iteration is the humbler, more general cousin.
"Value iteration reaches V* in finitely many steps." Cut. It converges asymptotically — the error is bounded by γk·e₀ and shrinks forever but never hits exact zero in finite arithmetic. The panel's error column shows it approaching, not arriving.
"Dynamic programming was named for its mathematics." Cut. By Bellman's own autobiography, he chose "dynamic programming" partly to disguise that he was doing mathematical research from a research-hostile administrator at RAND. The name is camouflage, not description.
"A larger discount converges faster." Kept, corrected. The contraction rate is γ itself, so γ closer to 1 converges slower — try γ = 0.99 in the panel and count the sweeps.
The red team's move: divide the update by γ, cancelling the discount. The V-dependent term now has coefficient 1 — the map is no longer a contraction, and the estimates diverge.
Cancel γ and the error stops shrinking, then blows up: the geometric bound is violated, the witness (window 7) recomputes, disagrees, and turns red. Nothing is faked; the attack is real and it is caught.