More parameters, same compute. A layer holds N small expert networks and one gate. The gate scores the input, a softmax turns the scores into weights, and only the top-k experts are ever run — the output is the renormalized, gate-weighted sum of just those k. The FLOPs track k, not N: that is how a model buys billions of parameters it does not have to compute. Terms go in, the router fires, the vector comes out. Blue builds it; red breaks it.
source Shazeer et al., Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017) — arxiv.org/abs/1701.06538. Rendered, not quoted.
The layer is four exact steps, no training required to decide them:
1 the gate maps the input to N logits, one per expert. 2 a softmax turns them into a distribution (sums to 1, all ≥ 0). 3 top-k keeps the k highest-weighted experts; the rest are never evaluated. 4 those k weights are renormalized to sum to 1, and the output is their weighted sum of the selected experts' outputs.
Live routing for the current input (fired experts in green):
| expert | gate p | fired | weight |
|---|
A dense transformer block runs one feed-forward network for every token. MoE replaces that single FFN with N of them and a gate — but still runs only k. Parameter count grows ×(N/k) while FLOPs stay flat.
That is the whole downstream story: the dense feed-forward sphere is this one's premise — MoE is what you get when you let the block hold more weight than it spends. The exact neighbouring slug is AMBER pending the FOLD index.
The blue team's live check: re-derive the gate softmax, confirm the router fired the k highest gate values, and count that exactly k experts were evaluated. If red reroutes, this badge is where it shows.
An input vector x ∈ ℝ⁴ meets a fixed bank of N experts and one gate, all seeded once (deterministic weights). Each expert is an affine map Wᵢx + bᵢ → ℝ³; the gate is a single matrix W_g x → ℝᴺ.
Pick an input and a sparsity k below. The gate has never seen a label — it just scores, and the softmax + top-k do the rest.
Every number below is computed on the spot from the seeded weights — the softmax, the selection, and the weighted sum are never looked up.
What the machine produces, proven: the gate is a real softmax, top-k selects exactly the k highest-scored experts, the output equals the renormalized gate-weighted sum of only those experts, and the evaluation count is k — not N. That last equality is the sparsity claim.
The blue team's witness (left) recomputes this live; the red team (right) tries to make the wrong experts run.
And the win is memory-bandwidth-bound, not FLOP-bound: the parameters still must be stored and shuffled across devices. "Same compute" is true for the matmuls and quietly false for the interconnect.
"An MoE with N experts is N× bigger for free." Cut. Capacity grows ×(N/k); the compute saving is real, but routing, replication, and comms overhead are not zero.
"The gate picks the single best expert." Cut. That is only k=1. For k≥2 the output is a renormalized blend of several — the engine computes exactly that sum.
"Unused experts cost nothing." Kept, corrected. They cost no FLOPs, but they cost memory and can go dead — hence the balancing loss the paper adds.
The red team's move: flip the router to select the bottom-k experts — the lowest gate scores — so the wrong experts run and the output changes. The blue team's witness (window 7) is watching the gate order.
Reroute to the least-favoured experts and the witness recomputes: the fired set is no longer the k highest gate values, so it disagrees with the reference and turns red. Nothing is faked; the attack is real and it is caught.