THE LAYER NORMALIZATION normalize a sample, not a batch
Batch norm needs a batch; layer norm needs only one sample. It normalizes across a single example’s features — (x − μ)/σ computed over that vector — so it works with batch size 1, in recurrent nets, and at inference without running statistics. That independence is exactly why it, not batch norm, became the normalization inside every transformer.
THE TECHNIQUE (x − μ)/σ across a sample's features
The demo normalizes one sample’s features [2,4,6,8] — mean 0, and it needs no other sample: live demo
HISTORY & CREDIT Ba, Kiros, Hinton · 2016
“Layer norm and batch norm are interchangeable.” — batch norm normalizes each feature across the batch; layer norm normalizes each sample across features — different axes, different behavior. cited
the axis · normalize across a sample’s features, not across the batch. the payoff · works at batch size 1, in RNNs, at inference — no running stats. 2016 · Ba, Kiros & Hinton — the transformer’s normalization.
A norm that needs no neighbours. normalization
RECOMMEND FOR I-13 batch size 1, on the compiler
On i-13, one sample [2,4,6,8] normalizes to mean 0 with batch size 1 — no other sample needed:
$ i13 run n2_layer-normalization.i13
RUN OK · 862 step(s)
mean=5 std=2.236 -> new_mean = 0
batch_size = 1 works_batch_1 = 1
Recommend as a NULL — a pinned normalization (B39). (x−μ)/σ over the feature axis is fixed; every correct implementation agrees. NULL — normalize a sample, not a batch.