The activation that woke the transformers up. ReLU snaps to zero and keeps a kink; GELU asks a softer question — weight each input by the chance a Gaussian would let it through. GELU(x) = x · Φ(x), where Φ is the standard-normal CDF. It is smooth everywhere, dips slightly negative for small negative x, and is the nonlinearity inside GPT and BERT. Down the center, a value goes in, the engine gates it, the activation comes out. The blue team builds and defends it; the red team tries to break it.
source Hendrycks & Gimpel, Gaussian Error Linear Units (GELUs) (2016) — arxiv.org/abs/1606.08415. Rendered, not quoted.
The exact definition is probabilistic: multiply the input by the probability a standard Gaussian draw is below it.
exact GELU(x) = x·Φ(x), Φ(x)=½(1+erf(x/√2)). tanh ≈ ½x(1+tanh(√(2/π)(x+0.044715x³))) — the fast form in real code. ReLU = max(0,x), the hard gate GELU smooths.
At the marked point x, the three faces and the exact slope GELU′(x)=Φ(x)+x·φ(x):
| quantity | value |
|---|
GELU is a smooth, probabilistic ReLU. Where the-first-neuron fires on a hard threshold — all-or-nothing above a bias — GELU replaces the step with a Gaussian gate: the input survives in proportion to how far past zero it lands.
That single change removes the kink. A smooth gate has a gradient everywhere, so the transformer's feed-forward block can be trained by descent without a dead corner at 0. Each sphere is the next one's premise: the threshold becomes the curve.
The blue team's live check: re-verify the defining properties — GELU(0)=0, GELU(x)→x for large +x, →0 for large −x, tanh-form within 1e-3, and the smooth negative dip GELU(−0.5)≈−0.154. If red swaps in ReLU, this badge is where it shows.
One real number x arrives — the pre-activation, a neuron's weighted sum before its nonlinearity. That is the whole input: a single value on the line, anywhere from strongly negative to strongly positive.
GELU asks one question of it: if I drew a standard Gaussian, what is the chance it falls below x? That probability Φ(x) becomes the gate. Feed x into the panel below and watch the gate open.
Exact: GELU(x)=x·Φ(x) with Φ from erf. The dashed curve is the tanh approximation used in production code.
Move x — every number is computed live from x·Φ(x) with erf, never looked up.
What the machine produces, proven: a value GELU(x) = x·Φ(x) that (i) is 0 at x=0, (ii) rises to ≈ x for large positive x, (iii) decays to ≈ 0 for large negative x, (iv) matches its tanh approximation to < 1e-3, and (v) is smooth — unlike ReLU it dips gently negative near zero (GELU(−0.5) ≈ −0.154, ReLU(−0.5) = 0) with a finite slope everywhere.
The blue team's witness (left) confirms these properties live; the red team (right) tries to make them false.
It is also not unique: SiLU/Swish x·σ(x), Mish, and GELU are near-identical smoothed gates — the paper's own experiments show small, dataset-dependent gaps. GELU won by defaults (BERT, GPT), not by proof. The negative dip that distinguishes it from ReLU is tiny; whether it matters is empirical, not settled.
"GELU is x times the sigmoid." Cut. That is SiLU/Swish, x·σ(x). GELU is x·Φ(x), the Gaussian CDF — close numerically, a different function.
"The tanh formula is the definition." Cut. It is an approximation (max error ≈ 4.7e-4 here). The definition is x·Φ(x); the tanh form is a fast stand-in.
"GELU always beats ReLU." Kept, corrected. It often helps and is smoother, but gains are small and task-dependent — the honest claim is "competitive and differentiable," computed, not "strictly better."
The red team's move: swap the smooth GELU for the hard ReLU = max(0,x) and hope no one notices. The blue team's witness (window 7) is watching the negative dip.
Swap in ReLU and GELU(−0.5) collapses from −0.154 to 0 — the smooth negative dip vanishes and the kink returns. The witness recomputes, sees the property fail, and turns red. Nothing is faked; the attack is real and it is caught.