THE LSTM — a dart

THE LSTM a memory that does not fade

The recurrent net that could finally remember. An LSTM carries a cell state down the sequence through a nearly-untouched channel — the constant error carousel — guarded by three gates: forget (what to erase), input (what to write), output (what to read). When the forget gate stays near 1, information (and its gradient) rides for hundreds of steps without vanishing. It powered sequence modeling for two decades before transformers.

THE TECHNIQUE cᵗ = f·cᵗ₋₁ + i·c̃

The demo runs the constant error carousel: a forget gate of 1 keeps the cell state intact for 50 steps: live demo

HISTORY & CREDIT Hochreiter & Schmidhuber · 1997

“LSTMs solved the vanishing gradient entirely.” — the carousel greatly mitigates it, but very long ranges still degrade; attention ultimately replaced it. cited

the carousel · the cell state flows down the sequence nearly untouched.
the gates · forget / input / output decide erase, write, read; forget ≈ 1 → memory (and gradient) survive.
1997 · Hochreiter & Schmidhuber (forget gate, Gers 2000).

A channel the gradient can ride for a long time. recurrence

RECOMMEND FOR I-13 the carousel, on the compiler

On i-13, forget gate 1 keeps the cell state at 100 across 50 steps; forget gate 0 resets it to 0:

$ i13 run n2_lstm.i13 RUN OK · 991 step(s) · call depth 51 forget=1: state 100 survives 50 steps forget=0: resets to 0 gates = 3

Recommend as a NULL — a gated recurrence (B39). The cell update is a pinned function of the gates; the carousel (like the residual identity) is a fixed structure. NULL — a memory that does not fade.