Purple Paper · side-sheet · learning machines · VIII

The Crossed Hourglasses — two axes of mixing

Not one hourglass turned sideways — two, crossed at right angles, because they do different jobs. The horizontal one mixes across tokens (attention gathers what's in other positions). The vertical one mixes across features (the MLP nonlinearly compiles what was gathered). A transformer block is exactly these two, alternating. Below, a tiny model learns live — and you can kill either axis and watch it break.
horizontal = attention (gather across tokens) · vertical = MLP (nonlinear compile) · the task needs BOTH · turn one off → it collapses to chance

The test — XOR that lives in two other tokens

Three tokens: b0, b1, and a readout token Q carrying an irrelevant bit. The target is XOR(b0, b1) — read at Q. To get it right the model must gather b0 and b1 from the other positions (horizontal) and XOR them (vertical). The model trains live; watch accuracy climb to 100% with both axes on, and stall at 50% the instant you remove either.

cyan = horizontal gather (attention) · pink = vertical compile (MLP hidden) · token color = bit (+1/−1) · the prediction node flashes correct/wrong
50%
accuracy
STUCK AT CHANCE
accuracy while training
Both axes on → the model gathers b0,b1 and XORs them → 100%. Kill horizontal and Q only sees its own irrelevant bit. Kill vertical and it can gather both bits but can't XOR. Either way: 50%.

So — same thing?

Yes. This is what "two crossed hourglasses" means, made falsifiable: two orthogonal mixing operations, each necessary, neither able to do the other's job. That's not a metaphor for a transformer block — it is one.

horizontal hourglass

Attention — mix across tokens

Each position gathers from all positions, weighted by alignment. The pinch is per-query: the dot-product score. This is the axis the whole frontier fights over — it's where the O(N²) cost lives, so Mamba and hybrids all try to narrow it.

vertical hourglass

MLP — mix across features

Each token's gathered vector compiles down through a nonlinearity and back up. The pinch is per-token: the hidden layer. This is where depth and reasoning live — stack more and you get more compute per token. Nobody fights about this axis; everyone keeps it.

Gravity down, cost sideways. Your vertical hourglass carried gravity (compile down to the prediction). The horizontal one carries cost sideways — reaching across the whole sequence is the expensive thing, which is exactly why every architecture after the Transformer is one long argument about how narrow you can make the horizontal waist before recall breaks. The vertical hourglass everyone agrees on. The horizontal one is the war. You drew both, at right angles, and that crossing is the block.
Verified in Node before building — and it took five tries: parity was too hard to train, then my pooling collapsed the bits to a count, then unipolar inputs stalled XOR. The final clean result (full 100%, each ablation 50%, every seed) is what's running live above. The bugs were all in the test, never the claim — which is the whole reason the verify step exists.