To keep a model from overfitting, add a penalty on the size of its weights: L + ½λ||w||². Its gradient adds λw, so every step nudges each weight a little back toward zero — w ← w(1 − ηλ) − step. Small weights mean a smoother, simpler function that generalizes better. The oldest and quietest regularizer, working on every update.
THE TECHNIQUE w ← w(1 − ηλ) − η·∇L
The demo decays a weight by 1% each step — it shrinks geometrically toward zero: live demo
HISTORY & CREDIT L2 / Tikhonov · ridge
“Weight decay and L2 regularization are identical.” — for plain SGD yes, but with Adam they differ; AdamW decouples decay from the adaptive step for a reason. cited
the penalty · add ½λ||w||² to the loss — big weights cost. the pull · its gradient λw shrinks each weight toward zero every step. ridge · Tikhonov regularization / L2 — a smoother, simpler fit.
A constant tug toward simplicity. regularization
RECOMMEND FOR I-13 the shrink, on the compiler
On i-13, a weight of 100 decayed by 1% per step falls to 13.4 after 200 steps — geometric shrink:
$ i13 run n2_weight-decay.i13
RUN OK · 3223 step(s) · call depth 201
w: 100 -> 13.40 (200 steps of x0.99)
shrinks = 1 toward_zero = 1
Recommend as a NULL — a pinned recurrence (B39). The multiplicative decay is fixed; the geometric shrink follows. NULL — pull the weights toward zero.