A model with two dimensions can carry three features — if they fire one at a time. Pack n feature directions into a d-dimensional space with n>d, keep the activations sparse, and each feature reads back clean. Down the center, data flows: a sparse code goes in, the frame packs it, the projection reads it out. The blue team builds the frame; the red team tries to break the packing.
source Elhage, Hume, Olsson, Schiefer, et al., Toy Models of Superposition (2022), Anthropic · Transformer Circuits Thread — transformer-circuits.pub/2022/toy_model. Rendered, not quoted.
The whole machine is three constants, chosen by hand — no training:
d = 2 ambient dimensions. n = 3 feature directions, placed as unit vectors at 0°, 120°, 240° — the cube roots of unity in the plane. Storage: h = W f (sum the active directions). Readout: x̂ = ReLU(Wᵀh) — project h back onto every direction and clamp the negatives.
Live geometry of the frame (all unit-norm, all pairwise cosines equal):
| pair | cos(vᵢ,vⱼ) | ‖vᵢ‖ |
|---|
If features were orthogonal, a unit could carry one feature and n would cap at d. Superposition breaks that: n>d directions share the space, so a single neuron lights up for many unrelated things at once.
That is why reading a network by neurons fails — the honest unit is the direction, not the axis. The next sphere makes it explicit: the linear representation — features are directions in activation space, here packed tighter than the space allows. This sphere is its overcrowded room.
The blue team's live check: re-measure the frame's cosines, confirm n>d, and re-run sparse vs dense recovery. If red forces the frame orthogonal, this badge is where it shows.
The input is a feature vector f of length 3. A sparse code has few nonzeros — the extreme is one-hot: exactly one feature active. That sparsity is the whole bargain: it is why three features fit in two dimensions.
| code | f | meaning |
|---|---|---|
| sparse | (1,0,0) | one feature — clean |
| 2-active | (1,1,0) | already interfering |
| dense | (1,1,1) | total cancellation |
Toggle the features in the panel below and feed the code into the frame.
Each active feature adds its unit direction. One active → the point lands on the ring; all three → they cancel to the origin.
| feature | in | recovered | ok |
|---|
Every number is computed on the spot from the frame — h = Wf, x̂ = ReLU(Wᵀh) — never looked up.
What the machine proves, in closed form: 3 features live in 2 dimensions. A single active feature is recovered exactly (error 0). A dense input cancels — the three 120° directions sum to the zero vector, so h=0 and nothing survives (error √3 ≈ 1.732).
The blue team's witness (left) confirms these numbers live; the red team (right) tries to make superposition impossible.
This toy is linear + one ReLU with a hand-built frame. Real models learn frames that are uneven — antipodal pairs, pentagons, "digons" — and features are not equally important. The clean −½ cosine here is a chosen ideal, not a claim about GPT.
"Each neuron encodes one concept." Cut. A direction encodes a feature; neurons are polysemantic because n>d packs features off the axes — the panel shows one point serving three directions.
"More features than dimensions is impossible." Cut. It is impossible to recover them all at once. Under sparsity you recover the active one exactly — capacity, bought with the assumption that few fire together.
"Superposition is lossless." Kept, corrected. It is lossy by design: interference is real (see the 2-active case, error ≈ 0.707). Sparsity keeps the loss off the features that matter.
The red team's move: force the frame orthogonal — keep only d=2 axis directions and drop the third feature. Now n = d and superposition is gone. The blue team's witness (window 7) is watching.
Force the frame orthogonal and only two features have a home — the "more features than dimensions" property fails, the witness recomputes, disagrees with the known frame, and turns red. Nothing is faked; the attack is real and it is caught.