bit-depth · how many features pack into m dimensions
The honest version of your bit-depth question. Sweep n_features as powers of two — 4, 8, 16, 32, 64 — into a fixed number of dimensions m, and measure how many features the model actually packs in, against sparsity. Dense (S=0): only m fit — one feature per dimension, no superposition. As sparsity rises, capacity climbs far past m: the dimensions start holding fractions of many features at once. "32 in 4 dims" isn't a constant — it's a function of sparsity.
m = 4 · press run
dense ceiling (no superposition)
m = 4 features
peak features packed
—
peak features / dimension
—
Reading the curves. Each line is a model with a different n_features (a power of two), all sharing the same m hidden dimensions, trained at each sparsity (constant importance, so this is pure packing — not importance ranking). The dashed white line is y = m: the dense ceiling, the most you can store with no superposition (one feature per dimension). Watch every curve sit at ≤ m on the left (dense) and climb above the dashed line as sparsity rises — that gap above the line is superposition, measured in features-per-dimension. So your instinct was right in spirit: this is a bit-depth / packing story — but the capacity lives in (features ÷ dimensions) × sparsity, not in the number 5. A from-scratch JS reimplementation built on Anthropic's Toy Models of Superposition (Elhage…Olah 2022, MIT) — math re-derived, trained live; the phenomenon is theirs.