One knob rules training: the learning rate, the size of each downhill step. Too small and learning crawls, wasting a thousand steps to inch to the bottom. Too big and each step OVERSHOOTS, bouncing higher and higher until it diverges to infinity. There’s a sharp stability edge between them. Getting it right is most of the art of training. Slide across the edge.
On a quadratic loss L=½a·x² (curvature a), gradient descent updates x ← x(1 − ηa). It converges only if |1 − ηa| < 1, i.e. 0 < η < 2/a — below that it converges (fastest near η=1/a), and ABOVE the bound each step overshoots and the magnitude GROWS geometrically to infinity. For a=2 (loss x²) the bound is η=1. This sharp edge is why learning-rate tuning, schedules, and warmup dominate practical training. A fail-loud self-check throws unless η=0.5 converges and η=1.1 diverges. ◆ real optimization, node-verified.
The exact bound 2/a is for a quadratic; real non-convex losses have varying local curvature, so practitioners use schedules, adaptive rates (Adam), and warmup rather than one fixed η. The stability threshold shown is exact.