Superposition is holographic storage — distributed, overlapping, over-capacity. The hidden space is the holographic volume: switch it between 3D and 4D. The model packs many features into it; each is a vector on the plate. Watch them pack past m dimensions into a rotating polytope — every axis carrying fragments of many features, like every patch of holographic film holds the whole scene. In 4D the constellation tumbles through two rotation planes at once and is projected 4D→3D→2D — the tesseract of features.
out = ReLU((x·W)·Wᵀ+b)) with m hidden dimensions (3 or 4), trained live (Adam, constant importance, sparse inputs). The arrows are the feature weight-vectors in that space, rendered as a rotating hologram-plate. At low sparsity it keeps only ~m (one per axis); as sparsity rises it packs far more than m into the volume, arranging them as a polytope and tolerating the overlap — that over-capacity is superposition, and it is literally distributed holographic storage (cf. Plate's Holographic Reduced Representations, ~1995). In 4D, points rotate in the (x,w) and (y,z) planes simultaneously, then project through the 4th dimension — far points shrink, the hallmark of a 4D→3D projection. Verified: 32 features pack ~19 into 4 dims at high sparsity. A from-scratch JS reimplementation built on Anthropic's Toy Models of Superposition (Elhage…Olah 2022, MIT) — math re-derived, trained live; the phenomenon is theirs.