Stack the layers too deep and the gradient dies on the way back — its magnitude a product of sub-unit factors, decaying geometrically toward zero. He, Zhang, Ren & Sun added one wire: each block computes y = f(x) + x. That + x puts an identity term in every Jacobian, so the gradient always has a road home. Down the center, data flows: the depth goes in, the engine differentiates a plain stack against a residual one, the surviving gradient comes out. The blue team builds it; the red team cuts the wire.
source K. He, X. Zhang, S. Ren, J. Sun, Deep Residual Learning for Image Recognition (2015) — arxiv.org/abs/1512.03385. Rendered, not quoted.
A network is a chain of blocks. Back-propagation multiplies each block's Jacobian, so the gradient reaching layer 1 is a product of L factors.
Plain block: y = f(x), so dy/dx = f'(x). For a sigmoid layer every factor has magnitude < 0.25 — multiply 50 of them and the product is ~10⁻³¹. The signal vanishes.
Residual block: y = f(x) + x, so dy/dx = f'(x) + 1. The + 1 is the identity path. Through L blocks the end-to-end Jacobian is ∏(f'ᵢ + 1) — it always contains the product of the ones, so it cannot collapse to zero.
The engine verifies the + 1 is real by finite difference: it perturbs a block's input and confirms the measured slope equals f'(x) + 1 to 1e-4. That identity term is the whole invention.
Two 2015 papers made very deep nets trainable, and they fix different failures. Batch-norm (Ioffe & Szegedy) re-scales activations so no factor blows up or crushes to zero — it keeps the gradient well-conditioned.
The skip connection keeps the gradient routed: even if f' is tiny, the + 1 carries it. Batch-norm conditions the residual branch; the identity path guarantees the through-line. Together they cleared 100 layers. Each sphere is the next one's premise.
The blue team's live check: recompute both stacks at depth 50, confirm plain has vanished (<1e-6) while residual is order 1 (>0.1), and re-verify the Jacobian's + 1 by finite difference. If red cuts the wire, this badge is where it shows.
Feed the machine one number: the depth L, how many blocks are stacked. Everything else is fixed and reproducible — a seeded pseudo-random set of weights (the same on every load), a sigmoid nonlinearity f, and a starting input x₀ = 0.5.
The engine then runs the exact back-prop chain rule twice: once through a plain stack (y = f(x)) and once through a residual stack (y = f(x) + x), and reports the magnitude of the gradient that reaches the first layer.
No training, no randomness at run time — the seed makes the whole experiment deterministic, which is what lets the witness check it.
Each residual block adds an identity path, so the gradient has a direct route home and cannot vanish. The plain stack multiplies L sub-unit factors and collapses.
| depth L | plain |∂| | residual |∂| |
|---|
Move the slider — both gradient magnitudes are computed live from the chain rule on the seeded weights, never looked up.
What the machine proves: at L = 50 the plain stack's gradient is … (below 1e-6 — dead) while the residual stack holds at … (above 0.1 — order 1). The per-block Jacobian is finite-difference-verified as f'(x) + 1. The identity path is why 100-layer nets train.
The blue team's witness (left) confirms these numbers live; the red team (right) cuts the wire to make them wrong.
The paper's own framing is worth stating: He et al. attribute the failure of plain deep nets primarily to a degradation problem (training error rises with depth), and note vanishing gradients were already largely handled by normalized init + BN. The clean "solves vanishing gradients" story this engine renders is the widely-accepted gradient-flow view of why the fix works — true and provable, but not the paper's headline claim. AMBER
"ResNets work because they fix vanishing gradients." Cut, corrected. That is the popular gloss; the paper targets the degradation problem. The identity path does preserve gradient flow — the engine shows exactly that — but it is one true consequence, not the sole stated cause.
"Deeper is always better." Cut. He et al.'s own 1202-layer CIFAR net overfit and scored worse than the 110-layer one. Depth buys nothing past the data's capacity.
"The + x is a cosmetic detail." Kept, corrected. Delete it in window 6 and the residual Jacobian falls back to f'(x) alone — the gradient vanishes just like the plain net. The wire is the mechanism.
The red team's move: remove the + x from every block, so each Jacobian is f'(x) alone instead of f'(x) + 1. The residual stack is now just a plain sigmoid stack — its gradient should die. The blue team's witness (window 7) is watching.
Cut the +x and the residual gradient at L=50 collapses below 1e-6 — the witness recomputes, disagrees with the known result, and turns red. Nothing is faked; the attack is real and it is caught.