ReLU’s hard corner throws away all negative signal and has an undefined kink. GELU smooths it: x·Φ(x), weighting each input by the probability a Gaussian falls below it. Small negatives leak through a little instead of dying, and the curve is smooth everywhere. It is the default activation in BERT, GPT, and most transformers — a gentler gate that trains a touch better.
THE TECHNIQUE GELU(x) = x·Φ(x)
The demo computes GELU (via the sigmoid approximation): 0 at 0, near-identity for large x, gently negative for small x<0: live demo
HISTORY & CREDIT Hendrycks & Gimpel · 2016
“GELU always beats ReLU.” — it helps in transformers; in many conv nets plain ReLU matches it at lower cost. cited
the weighting · multiply x by Φ(x), the Gaussian CDF — a soft, probabilistic gate. the leak · small negatives pass a little (unlike ReLU’s hard 0); smooth everywhere. 2016 · Hendrycks & Gimpel — BERT/GPT’s activation.
A gate with soft edges. activation
RECOMMEND FOR I-13 the smooth bend, on the compiler
On i-13 (sigmoid approx), GELU(0)=0, GELU(2)=1.94, GELU(−1)=−0.15 (a soft leak, not a hard zero):
$ i13 run n2_gelu.i13
RUN OK · 1988 step(s) · call depth 34
g0=0 g2=1.936 gneg=-0.154
smooth_negative = 1 near_identity_pos = 1
Recommend as a NULL — a pinned function (B39). GELU is a fixed activation; every correct implementation agrees. NULL — a smoother bend.