Plain gradient descent uses one step size for everything — too big for steep directions, too small for shallow ones. Adam gives EACH parameter its own adaptive step: it tracks a running average of the gradient (momentum) AND of its square (to normalize), so steep axes get gently damped and flat axes get boosted. It just works, which is why it’s the default optimizer training almost every neural network. Slide to descend and watch it glide to the bottom.
Adam (Adaptive Moment Estimation) keeps two running averages per parameter: the first moment m (like [[the-momentum|momentum]], a smoothed gradient) and the second moment v (a smoothed squared-gradient). It updates θ ← θ − lr·m̂/(√v̂+ε), where m̂,v̂ are bias-corrected — so each parameter gets its OWN effective step, large where gradients are small/consistent and damped where they’re large/noisy. This adaptivity makes it robust to sparse gradients and ill-scaling with little tuning, which is why it is the default optimizer for training deep networks. A fail-loud self-check throws unless it converges to the minimum of (x−5)². ◆ real optimization, node-verified.
Adaptive but not magic: Adam can generalize slightly worse than tuned SGD, may not converge on some convex problems (fixed by AMSGrad/AdamW), and still needs a base learning rate. The moment updates and bias correction are exact.