THE GRADIENT DESCENT — a dart

THE GRADIENT DESCENT roll downhill, one step at a time

Almost every neural network is trained by the same simple move: measure the slope of the error, and step the opposite way. x ← x − η·∇f. Repeat. The learning rate η sets the step; too big and you overshoot, too small and you crawl. Cauchy wrote it down in 1847 for astronomy; a century and a half later it is the engine under every model that learns.

THE TECHNIQUE x ← x − η·∇f

The demo minimizes f(x)=(x−3)² from x=0, η=0.1 — watch it roll to the minimum: live demo

HISTORY & CREDIT Cauchy · 1847

“Gradient descent finds the global minimum.” — it finds a local one; in the non-convex loss of a deep net there is no such guarantee (yet it works anyway). cited

the slope · ∇f points uphill toward steepest increase.
the step · go the other way, scaled by η: x ← x − η·∇f.
1847 · Augustin-Louis Cauchy — steepest descent; the workhorse of machine learning.

The whole of learning, in one repeated step. optimization

RECOMMEND FOR I-13 convergence, on the compiler

On i-13, 60 steps on (x−3)² from x=0 reach x=2.99999, the error crushed to ~2e−11:

$ i13 run nn_gradient-descent.i13 RUN OK · 1716 step(s) · call depth 61 xN = 2.999995 -- rolled to the minimum at 3 fN = 2.1e-11 converged = 1 decreased = 1

Recommend as a NULL — an iteration (B39/B40). The update is a pinned function of the gradient; convergence to a local min is an analytic fact, the iteration a resource cost. NULL — roll downhill, one step at a time.