Batch-norm steadies a feature by looking sideways at the whole batch — it needs a crowd. Layer-norm turns the axis ninety degrees: it steadies each token by looking across its own features, so a batch of one normalizes exactly as well as a batch of a thousand. Down the center, activations go in, each row is re-centered and re-scaled, the standardized rows come out. The blue team builds it; the red team tries to break it.
source Ba, Kiros & Hinton, Layer Normalization (2016) — arxiv.org/abs/1607.06450. Rendered, not quoted.
For one token's feature vector x (length H), compute its own mean and variance — no other token involved:
μ = mean(x), σ² = mean((x−μ)²), then
y = γ · (x − μ) / √(σ² + ε) + β
The raw per-token stats the panel is standardizing right now (these are what must become 0 and 1):
| token | raw μ | raw σ² |
|---|
Layer-norm is batch-norm with the normalization axis transposed. Batch-norm standardizes each feature across the batch; layer-norm standardizes each token across its features.
That single transpose buys everything: no running statistics, identical math at train and test, and it holds at batch size 1 — which is why the transformer, decoding one token at a time, uses this and not batch-norm. Each sphere is the next one's premise.
The blue team's live check: re-standardize the full matrix with γ=1, β=0 and confirm every row lands at μ≈0, σ²≈1 — and that a single-token batch does too. If red flips the axis, this badge is where it shows.
A block of activations: each row is a token, each column a feature (H = 6). These are raw pre-norm values on wildly different scales — the thing normalization exists to tame.
Layer-norm reads along each row. Batch-norm would read down each column. Same numbers — the whole difference is which way you look.
Full batch. Each row is normalized on its own — the count of tokens never enters the math.
| token | out μ | out σ |
|---|
Change γ, β, or the batch — every value is recomputed from x, μ and σ on the spot, never looked up.
The standardized block: with γ=1, β=0 every row has μ=0 and σ²=1; in general each row lands at μ=β and σ=|γ|, whatever the batch size. The current output:
And placement matters more than the formula: the paper's post-LN is hard to train deep without warmup; pre-LN trains but shifts the residual stream. "Add layer-norm" is never the whole decision — where it sits changes the gradients.
"Layer-norm normalizes over the whole layer, batch included." Cut. It touches only one example's features; the batch axis is never read — that is the entire point.
"Layer-norm and batch-norm are the same, just a different axis." Cut. Same formula, but BN keeps running statistics and couples examples; LN is per-example and stateless. The consequences diverge at batch 1.
"You must subtract the mean." Kept, corrected. RMSNorm drops μ and works fine — the essential operation is the rescaling, not the centering.
The red team's move: flip the axis — normalize down the columns (across the batch) like batch-norm. The formula is untouched; only the direction changes. The blue team's witness (window 7) is watching.
Normalize across the batch and a single token collapses (variance 0 — batch-norm needs a crowd) and per-row means stop being 0. The witness recomputes, disagrees, and turns red. Nothing is faked; the attack is real and it is caught.