◄ UD0   ← convergence   perceptron in silicon   → the deep neuron
Perceptron theory · II of III · the middling one

The capacity wall

Convergence tells you a perceptron will learn a separable pattern. This asks the harder question: how much can one hold? Feed it random labelled points and there's a hard ceiling — about two patterns per weight. Below it, almost any labelling is learnable. Above it, almost none. And the switch is sharp: a phase transition, the same kind of math as ice turning to water.
1965 Cover counts the separable labellings  →  1988 Gardner derives the capacity by statistical physics  →  the wall: α = N/d = 2
◐ WELL-SUPPORTED Cover (1965) · Gardner (1988). The result is solid and the curve below is exact (Cover's function-counting). The famous capacity αc = 2 from Gardner uses the replica method — a brilliant statistical-physics trick that was non-rigorous for decades (since proven in cases). Strong result; one of its two proofs is heuristic. Hence: middling, not bedrock.

The wall, exact

Take d weights and N random points, each given a random ±1 label (load factor α = N/d). Cover counted exactly what fraction of all possible labellings a perceptron can separate: P = 2·Σk=0..d−1 C(N−1, k) / 2N It's 1 for small α and crashes to 0 past the wall — passing through exactly 0.5 at α = 2. Drag d up and watch the slope sharpen into a vertical cliff: in the limit, a true phase transition.

x: load α = N/d (0→4) · y: fraction of labellings the perceptron can separate · amber line = the α=2 wall · violet dot = your probe
P(separable) at probe α
N = α·d patterns
capacity 2d (the wall)
At small d the wall is a gentle slope — small networks are forgiving. Crank d to 120 and the curve snaps to a near-vertical edge at α=2: a big perceptron is essentially binary — it stores everything up to 2d patterns and nothing past it. That sharpening is the thermodynamic limit, the signature of a real phase transition.

Why it's middling, said plainly

What's rock-solid: Cover's count is a finite combinatorial fact — the curve above is not a fit or a simulation, it's the exact formula. The capacity of 2d for random points in general position is proven.

Where it softens: (1) Gardner's elegant αc=2 derivation runs on the replica method, which assumes a symmetry that wasn't rigorously justified for decades (and only proven in special cases since). (2) It's capacity for random patterns — real data has structure, and structured patterns change the number. (3) The leap from "a perceptron stores 2d" to "this is why big neural nets generalize" is an active research frontier (double descent, the interpolation threshold) — suggestive, not settled. The wall is real; its reach into modern deep learning is the open part.