Gradient descent crawls down a narrow valley, zig-zagging across the steep walls. Give it inertia — let each step remember the last — and it rolls. Polyak's heavy ball: v ← β·v − α·∇f, then x ← x + v. On a strictly convex quadratic with a known minimizer, momentum reaches tolerance in far fewer steps than plain descent — computed live, both trajectories drawn. Terms go in, the ball rolls, the count comes out. Blue builds it; red over-spins it until it flies apart.
source B. T. Polyak, Some methods of speeding up the convergence of iteration methods, Zh. Vychisl. Mat. Mat. Fiz. 4:5 (1964) 791–803 · USSR Comput. Math. Math. Phys. 4:5, 1–17 — mathnet.ru/eng/zvmmf7713. Rendered, not quoted.
One extra state variable turns descent into a rolling mass:
v ← β·v − α·∇f(x) · x ← x + v
Unrolled, the velocity is an exponential moving average of every past gradient: vk = −α·Σ βk−j ∇fj. Recent gradients weigh most; old ones fade by β each step. β is the memory; α is the step.
Fed a constant gradient g, the velocity settles at the closed form −α·g/(1−β) — a geometric sum. For the current β the amplification is ·× the plain step. The witness re-derives this on load.
This is the-gradient-descent given a memory of its steps. Set β = 0 and the update collapses exactly back to it — momentum is a strict generalization, checked bit-for-bit in the witness.
Downstream, this same term is the first moment inside Adam, the reason SGD does not crawl, and the ancestor of Nesterov's accelerated gradient. Each sphere is the next one's premise.
The blue team's live check: re-run momentum on the canonical quadratic and confirm it converges to the known minimizer and beats gradient descent's step count. If red over-spins it, this badge is where it shows.
A strictly convex quadratic with a closed-form optimum:
f(x) = ½·xᵀA·x − bᵀx, A = diag(10, 1), ∇f(x) = A(x − x*).
Minimizer x* = [1, 1] (exact, A⁻¹b). Start x₀ = [4, 4]. The eigenvalues 10 and 1 make a long thin valley — condition number κ = 10 — the exact terrain where plain descent zig-zags and momentum pays off. This is what feeds the panel.
Both methods use the same α. Momentum adds only the β·v memory term — nothing else changes.
Every step, count, and trajectory is computed on the spot from A(x−x*) — never looked up.
What the machine proves, at the canonical setting (α = 0.10, β = 0.50): momentum reaches ‖x−x*‖ < 10⁻³ in 19 steps; plain gradient descent needs 76 — 4× faster, and both land on the same known optimum. At β = 0 the two are bit-identical. The counts below the canvas are recomputed for whatever α, β you set.
The blue team's witness (left) confirms convergence live; the red team (right) tries to make the ball diverge.
Momentum is also non-monotone: the loss can rise mid-roll (that overshoot is the mechanism). Too much β overshoots and slows; β ≥ 1 diverges outright — the panel lets you feel both. "Faster" is a tuned regime, not a free lunch.
"More momentum is always faster." Cut. Push β to 0.90 in the panel and the step count climbs back above plain GD — the ball overshoots and rings. The counts are live; watch them.
"Heavy ball gives Nesterov's optimal rate on any strongly convex function." Cut. True for quadratics; the Lessard et al. counterexample breaks it in general.
"Momentum and Nesterov are the same thing." Kept, corrected. Both carry velocity; Nesterov evaluates the gradient at the look-ahead point x + β·v, which is what buys the general guarantee.
The red team's move: set β = 1.5 — over-momentum. The velocity now compounds instead of averaging, and the iterates fly away from x*. The blue team's witness (window 7) is watching.
With β > 1 the geometric memory diverges: ‖x−x*‖ grows without bound. The witness recomputes, fails the convergence check, and turns red. Nothing is faked; the attack is real and it is caught.