THE ADAM — a dart

THE ADAM a learning rate for every weight

The default optimizer of deep learning. Adam keeps a running mean of the gradient (like momentum) and a running mean of its square, then steps by m / √v. The magic: dividing by √v makes the step size scale-invariant — roughly ±1 per weight regardless of how big or small that weight’s gradient is. Every parameter gets its own adaptive learning rate, for free.

THE TECHNIQUE step = m / (√v + ε)

The demo shows the step is ~1 whether the steady gradient is 2 or 100 — scale-invariant: live demo

HISTORY & CREDIT Kingma & Ba · 2014

“Adam always beats SGD.” — it converges faster but can generalize worse; well-tuned SGD+momentum often wins the final accuracy. cited

two moments · m = running mean of ∇, v = running mean of ∇².
normalize · step by m/√v — scale-invariant, ~±1 per weight, whatever the gradient magnitude.
2014 · Kingma & Ba — adaptive moment estimation.

Every weight, its own step size. optimizer

RECOMMEND FOR I-13 scale-invariance, on the compiler

On i-13, m/√v is 1 for a steady gradient of 2 and 1 for 100 — the step is independent of scale:

$ i13 run nn_adam.i13 RUN OK · 1596 step(s) · call depth 42 step(g=2) = 1 step(g=100) = 1 scale_invariant = 1 normalized = 1

Recommend as a NULL — a pinned update (B39). Adam’s moments and step are fixed functions of the gradient stream; the scale-invariance is an algebraic property. NULL — a learning rate for every weight.