A recurrent net forgets: run the gradient back through time and it shrinks by a factor every step, until the far past leaves no trace. Hochreiter & Schmidhuber cut a channel through the middle of the cell where the self-loop weight is fixed at 1 — the constant error carousel — so a stored value is carried exactly across a thousand steps. Down the center, a value goes in, sits through a long gap, and comes back out undamaged. The blue team builds the gates; the red team springs a leak.
source Hochreiter & Schmidhuber, Long Short-Term Memory, Neural Computation 9(8):1735–1780 (1997) — direct.mit.edu/neco/article/9/8/1735. Pre-arXiv; no clean archive.org id confirmed on search, so the journal of record is cited AMBER. Rendered, not quoted.
A memory cell holds a state c. Three gates, each a sigmoid squashed to (0,1), decide what happens to it each step:
| gate | value | controls |
|---|---|---|
| input i | σ(·) | how much new candidate g is written |
| forget f | σ(·) | how much of old c survives |
| output o | σ(·) | how much of c leaks out as h |
The update is c′ = f·c + i·g, and h = o·tanh(c′). Set f = 1, i = 0 and the cell becomes a straight wire: c′ = c. That fixed unit self-weight is the carousel — the original 1997 cell had no forget gate, just this permanent 1.
Left of here, the-first-neuron is a static unit: one input vector, one output, no yesterday. Stack it in time and the vanishing gradient erases the past.
The LSTM's answer is the gate: a learned valve that decides when the wire stays open. Right of here, attention (Vaswani, 2017) discards recurrence entirely and lets every step read every other step directly — a different fix for the same disease. This sphere is the hinge between them: memory by a protected loop, then memory by direct access.
The blue team's live check: re-run the carousel for 200 steps with the current forget gate and confirm the stored value is unchanged and the through-time gradient is exactly 1. If red springs a leak, this badge catches it.
Write a value into the cell at step 0 — the stored value c₀. Then close the input gate (i = 0) and let a long gap of N steps pass with no new information. The only question: is the value still there at the end?
For contrast the panel also runs a vanilla RNN memory, c′ = tanh(w·c) with |w| < 1 AMBER — the standard pre-LSTM recurrence and the standard illustrative baseline (w, tanh chosen for the demo). Feed the same c₀ into both and watch one hold while the other bleeds to zero.
Every number is computed on the spot by stepping the cell; nothing is looked up. The through-time gradient shown is dcN/dc₀ — the product of forget gates for the LSTM, the product of tanh′·w for the vanilla RNN.
Proven live: with f = 1, i = 0 the cell returns c₀ to within 1e-12 after 200 steps, and its through-time gradient is exactly 1 — no vanishing. The vanilla RNN with w = 0.9 falls below 0.01 within a few hundred steps and its gradient shrinks geometrically to a speck. One wire holds; the other forgets.
The blue team's witness (left) re-checks this every render; the red team (right) tries to make the "held" value leak.
And even at its best the LSTM is sequential: N steps mean N un-parallelizable hops. Attention (2017) threw out recurrence, read all positions at once, and won NLP outright. Gates also saturate and explode; the vanilla LSTM needed the forget-gate bias initialized high (Gers, 2000) to work at all. The carousel is a real fix — not the final one.
"The LSTM was born with a forget gate." Cut. The 1997 cell had only input and output gates and a fixed self-weight of 1. The forget gate is Gers, Schmidhuber & Cummins (1999–2000) — this panel's f = 1 is the original carousel.
"LSTM solved vanishing gradients, full stop." Cut. It solves it in the carousel's ideal limit; with learned gates in (0,1) the gradient still attenuates — the achievement is that it attenuates slowly enough to learn 1000-step lags.
"LSTM is obsolete." Kept, corrected. Transformers dominate, but gated recurrence lives on in streaming and low-latency work and in the state-space lineage (S4, Mamba). The mechanism outlived the hype cycle.
The red team's move: nudge the carousel's forget gate from 1 to 0.9. The wire is now a slow drain — the "preserved" value bleeds toward zero. The blue team's witness (window 7) is watching.
Set forget to 0.9 and the carousel's value decays like 0.9N instead of holding — the witness recomputes, finds c ≠ c₀ and the gradient ≠ 1, and turns red. Nothing is faked; the leak is real and it is caught.