THE GRADIENT CLIPPING — a dart

THE GRADIENT CLIPPING cap the step, keep the direction

The simplest cure for exploding gradients: if the gradient’s norm exceeds a threshold, rescale it down to that threshold — g ← g·(τ/||g||) when ||g|| > τ. The direction is preserved exactly; only the runaway magnitude is capped. One line that keeps recurrent and transformer training from detonating on a bad batch.

THE TECHNIQUE g ← g·τ/||g|| if ||g|| > τ

The demo clips a gradient of norm 100 down to 5 — the direction (3,4) is unchanged: live demo

HISTORY & CREDIT Pascanu, Mikolov, Bengio · 2013

“Clipping distorts the gradient direction.” — norm-clipping preserves the direction exactly; only per-element clipping (a cruder variant) can bend it. cited

the check · if ||g|| > τ, the step is too big.
the rescale · multiply by τ/||g|| — norm becomes τ, direction untouched.
2013 · Pascanu, Mikolov & Bengio — the standard exploding-gradient fix.

A leash on the magnitude, not the aim. optimizer

RECOMMEND FOR I-13 the clip, on the compiler

On i-13, a gradient (60,80) of norm 100 clips to (3,4) of norm 5, the direction preserved:

$ i13 run n2_gradient-clipping.i13 RUN OK · 1604 step(s) norm = 100 -> clipped (3,4) norm 5 capped = 1 direction_kept = 1

Recommend as a NULL — a pinned rescale (B39). The clipped gradient is a fixed function of g and τ; direction-preservation is algebra. NULL — cap the step, keep the direction.