Gradient descent reads the slope and inches downhill. Newton reads the curvature too — it fits a parabola to the surface and jumps to its bottom. On a convex quadratic that bottom is the true minimum, so Newton lands there in one step; on any smooth convex surface the error squares each step near the optimum. The blue team builds the update x ← x − H⁻¹g and proves it; the red team tears out the Hessian and watches it collapse into slow first-order descent.
source Newton (1685) / Raphson (1690); second-order optimization — Boyd & Vandenberghe, Convex Optimization (Cambridge, 2004), §9.5 web.stanford.edu/~boyd/cvxbook AMBER secondary. Rendered, not quoted.
Fit a quadratic to f at the current point and step to its minimum. That is the whole method:
x₁ = x₀ − H(x₀)⁻¹ ∇f(x₀)
H = ∇²f = the Hessian (curvature)
∇f = the gradient (slope)
Live decomposition of the step at the current iterate:
The Hessian inverse rescales the gradient by the local curvature — steep directions get short steps, flat directions long ones. That rescaling is exactly what a fixed learning rate cannot do.
the-gradient-descent takes x ← x − t∇f: one scalar step size t for every direction. Newton replaces that scalar with the matrix H⁻¹ — the gradient descent that finally reads the curvature.
The same shape recurs one level over: under Fisher scoring the Hessian is the expected Fisher information, and the-fisher-information's natural gradient is Newton in the metric that geometry defines. Each sphere is the next one's premise: slope → curvature → information-geometry.
The blue team's live check: recompute the three load-bearing facts — one step is exact on the quadratic, the error squares on the non-quadratic, and the Hessian is SPD at the optimum. If red swaps the Hessian for the identity, this badge flips.
Two constructed convex problems with closed-form answers, so every claim is checkable to 1e−9:
A. strictly-convex quadratic (2D)
f(x) = ½ xᵀAx − bᵀx, A = [[4,1],[1,3]], b = (1,2)
∇f = Ax − b, ∇²f = A (constant, SPD)
minimizer x* = A⁻¹b = (1/11, 7/11)
B. non-quadratic convex (1D)
f(x) = eˣ − x, ∇f = eˣ − 1, ∇²f = eˣ
minimizer x* = 0, f″(0) = 1 > 0
A is exactly quadratic, so Newton's parabola is f — one step is exact. B is not, so the parabola only approximates f, and the error decays quadratically. Feed either into the panel below.
| k | error eₖ | eₖ / eₖ₋₁² |
|---|
Every number is computed on the spot: the step solves H·step = ∇f live, never a lookup. Watch the ratio column sit near 0.5 — the signature of quadratic convergence.
What the machine proves, checked at boot:
• On the quadratic, Newton reaches x* = (1/11, 7/11) in a single step from any start — H⁻¹ cancels the curvature exactly.
• On eˣ−x, the error squares each step: eₖ₊₁ ≈ ½ eₖ².
The blue witness (left) re-checks these live; the red team (right) tries to make them false.
And it is affine-invariant but expensive: forming and inverting the Hessian costs O(n³) time and O(n²) memory each step. For a model with millions of parameters that is impossible — which is why deep learning runs first-order (SGD/Adam) or quasi-Newton (L-BFGS, K-FAC) approximations, never the exact Hessian.
"Newton's method always converges." Cut. Only locally, and quadratically only near the optimum. From a bad start pure Newton can diverge; global convergence needs damping / backtracking line search.
"Newton's method finds roots of f." Kept, corrected. That is the root-finder x ← x − f/f′. The optimizer applies it to the gradient: x ← x − H⁻¹∇f — roots of ∇f, one derivative up.
"Second order always beats first order." Cut. Per step it costs O(n³). At scale, cheap first-order steps win on wall-clock — fewer iterations does not mean less time.
The red team's move: replace the Hessian H with the identity — turning Newton back into plain gradient descent (step 1). The curvature rescaling is gone, so the one-step-exact property dies. The witness (window 7) is watching.
Swap in the identity and Newton no longer lands on the quadratic's minimum in one step — the witness recomputes, the one-step error jumps off zero, and the badge turns red. Nothing is faked; the attack is real and it is caught.