Plain gradient descent zig-zags across ravines and crawls along shallow slopes. Momentum fixes both by giving the update memory: keep a running velocity, add each gradient to it, and step by the velocity — v ← βv + ∇f, x ← x − ηv. Consistent directions accumulate speed (up to 1/(1−β)×), oscillations cancel. A heavy ball rolling downhill instead of a cautious walker.
THE TECHNIQUE v ← βv + ∇f
The demo accumulates a steady gradient with β=0.9 — velocity builds to 10× (1/(1−β)): live demo
HISTORY & CREDIT Polyak · 1964
“Momentum just uses a bigger learning rate.” — it accumulates consistent directions while cancelling oscillating ones; a bigger η amplifies both. cited
the velocity · v ← βv + ∇f — the gradient with a memory. terminal speed · a steady gradient accumulates to 1/(1−β)× — 10× at β=0.9. 1964 · Boris Polyak — the heavy-ball method.
A ball with inertia, not a cautious step. optimizer
RECOMMEND FOR I-13 terminal velocity, on the compiler
On i-13, a steady gradient of 1 with β=0.9 builds velocity to 9.999 — the 10× terminal speed:
$ i13 run nn_momentum.i13
RUN OK · 2003 step(s) · call depth 91
v_final = 9.999 -- -> 1/(1-0.9) = 10
amplifies = 1 near_terminal = 1
Recommend as a NULL — a pinned recurrence (B39). The velocity is a fixed function of the gradient history; the 1/(1−β) limit is a geometric-series fact. NULL — give the gradient inertia.