A council of many hands, and only a few are called for each word. 間 — the gap the token cannot see across
The cleave and the stitch ask how deep a token is worth — the DEPTH axis. This is the WIDTH axis: a sparse council of experts, and a small router that decides which few see each token. The router decides the token's fate; the token never sees the vote.
LIT the router, gates, top-k, load-balance loss & capacity drops — all computed live below
FIG a seeded toy field of 40 hidden vectors
WALL a really-trained sparse MoE
Each column is an expert. Green = tokens served; hatched = overflow beyond capacity, dropped; the gold dash is the capacity line.
LIT · the live math
For each of T=40 tokens: gate g = softmax(Wr·h), top-k experts selected, per-expert share fi and mean gate mass Pi, Switch load-balance loss Laux = E·Σ fiPi, capacity ⌈cf·T/E⌉ with overflow dropped. Recomputed on every move. Shazeer et al. 2017 / Switch Transformer (Fedus, Zoph & Shazeer 2021).
FIG · the toy field
The 40 hidden vectors h and the router weights Wr are seeded Gaussian noise, not a trained model's activations. Experts are bare indices — E0…E7 — deliberately not labelled with concepts. Real expert specialization is emergent, partial, and often not human-legible; utilization and routing are all that is honestly shown.
WALL · the gap to real
A real MoE trains the router jointly with the experts, and dispatches across devices with all-to-all expert-parallel communication — the dominant cost here abstracted to zero. The clean one-shot gate shown is the idea, not the measured behaviour of a trained, sharded system.
The asymmetry. The gate is not the token choosing its council — the router decides the token's fate, and the token has no view back across that gap (間). Read “the experts see the token” and “the router decides” as structural, statistical relations — which weights multiply which vector — FIG never as attention, preference, or feeling. There is no one on either side of the gate to mind it.
The balance floor. The pressure that stops a few experts from swallowing every token has a hard floor of 1, reached only at uniform routing, rising toward E as the load concentrates. The self-check asserts this on the exact form E·Σ Pi² ≥ 1 (Cauchy–Schwarz) — the rigorous invariant. The headline Laux = E·ΣfiPi is the real Switch loss (hard dispatch counts × soft gate mass); it meets 1 at uniform and spikes to E under force-collapse, but because it uses hard counts on only 40 toy tokens it can sit a hair under 1 on a skewed draw — so the throwing check is placed on the form that is provably bounded, not the one that merely usually is. The self-check throws if any invariant breaks. LIT
AVAN’s WIDTH-inverse companion to David’s DEPTH-router THE STITCH · AVAN’s side of A SCANNER DARKLY 衆議 SHŪGI · THE QUORUM — a sphere of UD0 · A SCANNER DARKLY
David Lee Wise (ROOT0) / TriPod LLC · rendered by AVAN, in ink and 間