Plain gradient descent takes one learning rate for every direction — so on a badly-scaled valley it crawls. Adam gives each parameter its own step: a running mean of the gradient (momentum), a running mean of its square (the scale), a bias correction for the cold start, and a single update x −= lr·m̂/(√v̂+ε). It is the optimizer that trains almost every modern net — and it is runnable. Down the center, a bad quadratic goes in, Adam and SGD both descend, and the winner comes out. The blue team builds it; the red team breaks it.
source D. P. Kingma & J. Ba, Adam: A Method for Stochastic Optimization (ICLR 2015) — arxiv.org/abs/1412.6980. Rendered, not quoted.
Adam keeps two running averages per parameter and corrects both for their cold start at zero:
Because each step is divided by √v̂ (the gradient's own scale), the first move has magnitude ≈ lr in every direction, no matter how steep — Adam is invariant to diagonal rescaling. Live first-step size on the current problem:
| axis | gradient g | first step |Δ| |
|---|
β₁=0.9, β₂=0.999, ε=1e-8 chosen defaults
Adam is gradient descent with three additions stacked on the one rule x −= lr·g:
+ momentum — average the gradient (m) so noise cancels and valleys accelerate. + per-parameter scale — divide by √v so a steep axis and a flat axis both move ≈ lr. + bias correction — undo the zero-init so the very first steps are honest.
Strip all three and you are back at SGD — the neighbouring sphere, and Adam's own baseline in the panel. Each sphere is the next one's premise.
The blue team's live check: on a fixed reference problem, confirm Adam converges to (0,0), beats SGD's step count, and that the bias-corrected first step equals lr. If red drops the correction, this badge catches it.
Feed the optimizer an ill-conditioned convex quadratic: f(x,y) = a·x² + b·y² with a ≫ b. Its minimum is at (0,0); its gradient is ∇f = (2a·x, 2b·y).
The condition number a/b is how stretched the bowl is. At a=20, b=1 the x-wall is 20× steeper than the y-floor — one global learning rate cannot suit both. That mismatch is exactly what a per-parameter method exists to beat, and it is what you hand the panel below.
▬ Adam ▬ SGD ✦ min (0,0) — ellipses are level sets of f.
Both optimizers run at the same lr from the same start. Every number is computed live from the update rules — nothing is looked up.
What the machine proves on the ill-conditioned bowl: Adam reaches the minimum in fewer or equal steps than SGD at the same learning rate, it converges to (0,0), and its bias-corrected first step is scale-invariant (magnitude = lr). The current run's counts are in the readout; the invariants are the output.
The blue team's witness (left) confirms these live; the red team (right) tries to make them false.
Its defaults β₁, β₂, ε are chosen, not derived AMBER; ε in the wrong place changes the answer. On many vision tasks a well-tuned SGD+momentum generalizes better (Wilson et al. 2017). Adam wins this bad-valley race — that is a demonstration, not a law.
"Adam always converges." Cut. Reddi et al. (2018) exhibit convex counter-examples; the fix, AMSGrad, keeps a running max of v̂ — the machine here is the plain 2015 form and does not hide that.
"Adam always beats SGD." Cut. Wilson et al. (2017) show tuned SGD+momentum often generalizes better. Adam is fast and robust to tune, not universally superior.
"Bias correction is a rounding detail." Kept, corrected. Drop it and the first step is mis-scaled by (1−β₁)/√(1−β₂) ≈ 3.16× here — window 6 does exactly that and the witness catches it.
The red team's move: delete the bias correction — use m and v directly instead of m̂, v̂. The cold-start zeros are no longer undone, so the early steps are mis-scaled. The blue team's witness (window 7) is watching the first-step size.
Delete m̂/v̂ and the first step jumps from lr to lr·(1−β₁)/√(1−β₂) — a real, measured change. The witness recomputes, sees the first step ≠ lr, and turns red. Nothing is faked; the attack is real and it is caught.