The iconic experiment, reproduced live. A tiny autoencoder must store 5 features in only 2 dimensions. When features are dense, it can only keep the 2 most important — the rest collapse to zero. As you raise sparsity, it starts packing all five into the plane, overlapping them and tolerating interference: superposition. Watch the feature vectors organize from the axes into a pentagon.
out = ReLU((x·W)·Wᵀ + b) — it compresses x (5 sparse features) through 2 hidden dims and tries to reconstruct it, weighting each feature by an importance that decays 1 → 0.59. With dense inputs (S=0) it behaves like PCA: keep the top 2, drop the rest (you'll see 2 arrows on near-orthogonal axes, 3 collapsed to the origin). As sparsity rises, collisions become rare, so storing an extra feature — even with interference — pays off: the arrows fan out into a digon → triangle → pentagon. That packing of more features than dimensions is superposition — and it's why the inside of a network looks like a smear: more meanings than neurons. This is a from-scratch JS reimplementation of the experiment in Anthropic's Toy Models of Superposition notebook (Elhage, … Olah, 2022, MIT) — the math is re-derived and trained live; the result is theirs.