THE EXPLODING GRADIENT — a dart

THE EXPLODING GRADIENT the vanishing gradient's evil twin

Backprop multiplies a gradient through every layer — and if those factors are larger than 1, the product races to infinity instead of zero. (4)¹⁰ ≈ 10⁶. The weights blow up to NaN and training dies. It plagues recurrent nets especially, where the same weight multiplies at every timestep. The fix is blunt and effective: clip the gradient before it explodes.

THE TECHNIQUE ∏ large factors → ∞

The demo multiplies a factor of 4 through 10 layers — it explodes past a million, while 1 stays stable: live demo

HISTORY & CREDIT Hochreiter · the RNN wall

“Exploding and vanishing gradients are opposite problems needing opposite fixes.” — both come from the same product of factors; the cures (clipping, gating, residuals, init) overlap. cited

the product · each layer’s factor multiplies the gradient.
the blow-up · factors > 1 over many layers → ∞ → NaN; recurrent nets suffer most.
the fix · gradient clipping (next dart) — cap the norm before it detonates.

Depth multiplies until the signal detonates. the wall

RECOMMEND FOR I-13 the blow-up, on the compiler

On i-13, 4¹⁰ = 1,048,576 (explodes) while 1¹⁰ = 1 (stable):

$ i13 run n2_exploding-gradient.i13 RUN OK · 313 step(s) explode = 1048576 explodes = 1 mild = 1 need_clip = 1

Recommend as a NULL — a product identity (B39). The gradient product is fixed arithmetic; the blow-up is its analytic consequence. NULL — the vanishing gradient’s evil twin.