THE RESIDUAL CONNECTION — a dart

THE RESIDUAL CONNECTION add the input back

The one-line change that let networks go from tens of layers to hundreds: instead of a block computing f(x), have it compute x + f(x) — add the input straight back. Now a block only has to learn the residual, the correction, and the gradient gets a clean identity path home: ∂(x+f)/∂x = 1 + f′ — the +1 keeps it from collapsing to zero even when f′ vanishes. ResNet stacked 152 layers and won; every deep architecture since uses it.

THE TECHNIQUE y = x + f(x); ∂y/∂x = 1 + f′

The demo shows the gradient through a residual block stays 1 even when f’s gradient vanishes — 50 layers deep: live demo

HISTORY & CREDIT He et al. · 2015

“Residual connections just add more parameters.” — the skip has zero parameters; its whole power is the identity path that carries the gradient. cited

the skip · compute x + f(x) — learn only the correction.
the highway · ∂(x+f)/∂x = 1 + f′ — the +1 identity term keeps the gradient from vanishing to zero (it becomes 1, not 0, when f’ → 0).
2015 · He et al., ResNet — 152 layers, and depth was solved.

An identity highway for the gradient. architecture

RECOMMEND FOR I-13 the identity path, on the compiler

On i-13, when f’s gradient is 0 the residual gradient is still 1 (the plain net’s is 0), and it survives 50 layers:

$ i13 run nn_residual-connection.i13 RUN OK · 681 step(s) · call depth 51 grad_residual = 1 grad_plain = 0 deep_grad(50) = 1 trains_deep = 1

Recommend as a NULL — an identity path (B39). The residual gradient 1+f′ is fixed calculus; the identity skip preserves it. A structural choice, not a same-function DOF. NULL — add the input back.