THE UNIVERSAL APPROXIMATION — a dart

THE UNIVERSAL APPROXIMATION one hidden layer is enough

A startling theorem: a neural network with a single hidden layer of enough neurons can approximate any continuous function to any accuracy. The intuition is simple — each pair of neurons builds a little bump, and enough bumps trace any curve. It says nothing about how many neurons or how to find the weights (depth wins in practice), but it settles the question: nothing is beyond reach in principle.

THE TECHNIQUE a bump per neuron pair → any function

The demo builds a localized bump from three ReLUs — the atom of universal approximation: live demo

HISTORY & CREDIT Cybenko 1989 · Hornik 1991

“Universal approximation means shallow nets are enough.” — it guarantees existence, not efficiency; deep nets can need exponentially fewer neurons for the same function. cited

the bump · a few neurons make a localized hump; ReLU(x)−2ReLU(x−1)+ReLU(x−2) is a triangle.
the sum · enough bumps trace any continuous curve, to any ε.
1989/91 · Cybenko (sigmoid), Hornik (general) — existence, not efficiency.

Any function, in reach of one wide layer. theorem

RECOMMEND FOR I-13 the bump, on the compiler

On i-13, ReLU(x)−2ReLU(x−1)+ReLU(x−2) is a bump: 0 at x=0, 1 at x=1, 0 at x=2:

$ i13 run nn_universal-approximation.i13 RUN OK · 142 step(s) · call depth 2 bump(0)=0 bump(1)=1 bump(2)=0 is_bump = 1 can_approximate = 1

Recommend as a NULL — an existence theorem (B39). The approximation guarantee is a pinned mathematical fact; the bump construction is fixed. NULL — one hidden layer is enough.