Learn by fitting the data you have, and hope it holds on the data you do not. Over a finite hypothesis class, ERM returns the rule with the smallest training error — and the whole theory is one question: how far can that empirical risk drift from the true risk? Down the center, data flows: the labeled sample goes in, ERM picks a hypothesis, the two risks come out. The blue team builds and bounds it; the red team lets capacity run wild and breaks it.
source Vapnik, The Nature of Statistical Learning Theory (Springer, 1995) — the ERM principle. link.springer.com/book/10.1007/978-1-4757-3264-1. No stable open id — cited by author/title/year, marked AMBER. Rendered, not quoted.
ERM is one line: pick the hypothesis that fits the sample best.
ĥ = argminh∈H Remp(h), Remp(h) = (1/m) Σ 1[h(xi) ≠ yi].
Nothing is looked up: for the live sample, every threshold in H is scored and the minimum is taken (ties → smallest threshold). Live ranking:
| hypothesis | Remp | R (true) |
|---|
Every trained model runs this loop. Maximum likelihood is ERM in disguise: maximizing log-likelihood is minimizing the average negative-log-likelihood loss — empirical risk under a different loss.
The recast is only safe when capacity is controlled. Vapnik’s move was to make that condition exact: ERM is consistent iff the class has finite VC dimension, so uniform convergence holds. That premise is the next sphere.
The blue team’s live guard: re-run ERM on a fixed sample and check the selected hypothesis’s gap against the finite-class bound. If red swaps in an unbounded class, this badge is where it shows.
The world is a domain X = {0 … 255} with a uniform distribution, and a target concept y (here a threshold: y(x)=1 iff x ≥ t*). We never see the whole world — only a sample of m labeled points drawn i.i.d.
The learner is a finite hypothesis class H: the 257 threshold rules [x ≥ t], t = 0…256. Each h has:
| quantity | definition | known? |
|---|---|---|
| Remp(h) | error rate on the m sampled points | yes |
| R(h) | error rate over all 256 points | hidden |
ERM can only see the left column. The theory is about how well minimizing it controls the right one. That is what you feed the panel below.
Realizable: t* is itself in H, so a zero-error rule exists — but a small sample rarely pins it exactly, and that slack is the true risk.
Change any control — both risks and the bound are recomputed from the four rules on the spot, never looked up.
What the machine proves, live: ERM returns exactly the minimum-training-error hypothesis; for any fixed h the empirical risk is an unbiased estimate of the true risk (E[Remp]=R); in the realizable case ERM hits zero training error; and the selected rule’s gap |Remp−R| stays inside the finite-class uniform bound.
The blue witness (left) confirms the guarantee live; the red team (right) removes the one premise that makes it true.
Real failure modes: broken i.i.d. (distribution shift), a class rich enough to interpolate noise, and label noise that makes zero training error a symptom rather than success. ERM is the principle; capacity control is the theorem.
“Lower training error is always better.” Cut. Only until capacity outruns the sample; past that, driving Remp down raises the gap. The panel shows Remp=0 with true risk still positive.
“Zero training error means you learned the concept.” Cut. A lookup table gets Remp=0 and ~50% true error. Window 6 does exactly this.
“ERM needs no assumptions.” Kept, corrected. It needs i.i.d. sampling and a class of controlled capacity — Vapnik’s finite-VC condition is what turns fitting into learning.
The red team’s move: swap the finite class for an unbounded one that can memorize the labels — Remp drops to 0 while the true risk stays high. The finite-class bound no longer bounds anything; the witness (window 7) is watching.
Switch classes and Remp=0 buys nothing: the witness recomputes the selected hypothesis’s gap, finds it far above the bound, and turns red. Nothing is faked — the attack is real and it is caught.