Stand anywhere on a hill in fog. Feel the ground's slope with your feet, and step downhill. Repeat. In 1847 Cauchy wrote that rule as arithmetic — (x,y) ← (x,y) − η·∇f — and it is runnable: on a convex bowl the loss falls every step to the true minimum, while one wrong knob (a step too big) sends it flying. Down the center, data flows: the surface and a start point go in, the engine steps, the descending trace comes out. The blue team builds and defends it; the red team tries to break it.
source A.-L. Cauchy, Méthode générale pour la résolution des systèmes d'équations simultanées, Comptes rendus Acad. Sci. Paris 25 (1847), p. 536 — in vol. 25 at archive.org/details/comptesrendusheb25acad. Rendered, not quoted.
The gradient ∇f points uphill, fastest; its negative points downhill, fastest. Take a step of size η against it: x ← x − η∂f/∂x. On the bowl f = a·x² + b·y² the update is per-axis linear, x ← x·(1 − 2aη), so it converges only while each factor's size is under 1.
For the current η, the two contraction factors (want |·| < 1):
| axis | curvature | factor 1−2·η | stable? |
|---|
Cauchy's 1847 half-page is the ancestor of nearly every fitting procedure since. Least squares descends a quadratic error; k-means descends within-cluster variance by alternating steps; backpropagation is exactly this update with ∇f supplied by the chain rule through a deep network.
One deletion made it the engine of modern AI: don't solve for the minimum in closed form — step toward it. That scales to billions of parameters where no closed form exists. The next sphere in the arc feeds this gradient backward through the layers: THE FOLD ▸ the backward pass. Each sphere is the next one's premise.
The blue team's live re-check: re-run the descent from scratch and confirm the bowl run falls to the minimum, a too-large η diverges, and the analytic gradient equals a finite-difference probe. If red tampers, this badge is where it shows.
Gradient descent needs three inputs: a differentiable loss f(x,y), a start point, and a learning rate η. From the loss it derives one more thing analytically — the gradient:
| surface | f(x,y) | ∇f = (∂/∂x, ∂/∂y) | minimum |
|---|---|---|---|
| bowl (convex) | a·x² + b·y² | (2a·x, 2b·y) | (0, 0) |
| Rosenbrock | (1−x)² + 100(y−x²)² | see engine | (1, 1) |
The bowl uses a=1, b=3 — an AMBER choice: a mild ellipse (condition number 3) so the zig-zag is visible without being pathological. Feed these into the panel below.
Click the surface to drop a new start point.
| step | x | y | f(x,y) |
|---|
Every point on the trace is computed live: step = current point minus η times the analytic gradient. Nothing is looked up.
What the machine proves, live: on the convex bowl with a stable η the loss decreases every step and the point converges to the true minimum (0,0) within tolerance; the analytic gradient matches a finite-difference probe to ~1e−6; and a step size above the threshold η* = 0.333 makes the loss diverge instead. All three are re-checked by the witness on the left.
The blue team's witness (left) confirms these live; the red team (right) tries to make the descent lie.
And the step size is fragile: too small crawls, too big diverges (the slider proves it), and the safe band shrinks as the surface gets more ill-conditioned. "Gradient descent works" is a claim about convex, well-conditioned problems — everything real needs momentum, adaptivity, or luck in the initialisation.
"Gradient descent finds the global minimum." Cut. Only for convex f. On the Rosenbrock demo it stalls in the valley; the bowl is convex, so there it does reach the one true minimum.
"A bigger learning rate always trains faster." Cut. Above η* = 1/max(a,b) = 0.333 the loss increases — the divergence you can trigger on the slider right now.
"Cauchy invented backpropagation." Kept, corrected. Cauchy gave the descent step (1847). Efficient reverse-mode gradients are Linnainmaa (1970); backprop for nets is Rumelhart–Hinton–Williams (1986). His is the step, not the chain rule.
The red team's move: flip the update's sign so it reads x ← x + η∇f — ascent, climbing away from the minimum. The blue team's witness (window 7) is watching.
Flip the sign and the descent becomes an ascent: the loss rises instead of falling. The witness recomputes the bowl run, finds it no longer converges, and turns red. Nothing is faked; the attack is real and it is caught.