Starting training at full learning rate can blow up a fresh, badly-scaled network. Warmup ramps the rate up gently from near zero over the first few thousand steps, letting the statistics settle, then decays it for fine convergence. The transformer’s schedule — lr ∝ min(step⁻⁰·⁵, step·warmup⁻¹·⁵) — peaks exactly at the warmup point. A gentle start that made large transformers trainable.
THE TECHNIQUE ramp 0→peak over warmup, then decay
The demo shows the rate ramping up to the warmup step, then decaying — peaking at warmup: live demo
HISTORY & CREDIT the transformer schedule · 2017
“Warmup is a minor tuning trick.” — for large transformers and Adam it is often essential; skipping it can diverge in the first hundred steps. cited
the ramp · rate rises from ~0 over the warmup steps — let the statistics settle. the decay · after the peak, the rate falls for fine convergence. 2017 · the transformer schedule — peak exactly at the warmup point.
A gentle push before the long coast. schedule
RECOMMEND FOR I-13 the schedule, on the compiler
On i-13 (warmup 10), the rate is 20 at step 2 (ramping), 100 at step 10 (peak), 20 at step 50 (decaying):
$ i13 run n2_learning-rate-warmup.i13
RUN OK · 60 step(s)
lr(2)=20 (ramp) lr(10)=100 (peak) lr(50)=20 (decay)
warmup_then_decay = 1
Recommend as a NULL — a pinned schedule (B39). The learning-rate curve is a fixed function of the step. NULL — ease in, then coast down.