A perceptron finds a line between two classes. Vapnik asked the harder question: of the infinitely many lines that separate them, which one generalizes? His answer is geometric and provable — the one that leaves the widest empty street, maximally far from the nearest point of either class. The points touching the kerb are the support vectors; they alone define the boundary. Down the center, data flows: the points go in, the solver finds the widest street, the classifier comes out. The blue team builds and defends it; the red team tries to break it.
source V. Vapnik, The Nature of Statistical Learning Theory (1995); the SVM after Cortes & Vapnik, Support-Vector Networks, Machine Learning 20 (1995) 273–297 — archive.org/details/natureofstatisti0037vapn. Rendered, not quoted.
A line is w·x = b. Rescale so the nearest point of each class sits at w·x − b = ±1; then the street has half-width 1/‖w‖. The classifier that generalizes best is the one that maximizes that width, i.e. minimizes ‖w‖ subject to every point being on its own side by at least 1.
Equivalently: the max margin is half the distance between the two classes' convex hulls. The solver here finds that width exactly (candidate normals = point-to-point and edge-perpendicular directions), so nothing is approximate. Live, for the data below:
Rosenblatt's perceptron (1958) halts the instant it finds any separating line — its answer depends on the order it saw the data. Two perceptrons, same points, different lines.
Vapnik's move is to make the choice canonical: among all separators, take the widest-margin one. That single extra demand ties the boundary to a generalization bound — the wider the street, the smaller the capacity, the tighter the guarantee on unseen data (VC theory). The perceptron finds a wall; the margin picks the wall. Each sphere is the next one's premise — the perceptron hands its arbitrary line here to be made optimal.
The blue team's live check: recompute the street width, confirm it beats every candidate direction and 1500 random separating lines, and confirm the support vectors sit exactly on the kerb. If red tampers, this badge is where it shows.
16 labelled points in the plane — 8 red and 8 blue — drawn from a fixed-seed generator that rejects anything inside a hidden gap, so the two classes are linearly separable by construction. That is the whole input: coordinates and a label. No line is supplied; the machine must find the best one.
The generator is deterministic (seed 0x5EED11) so the panel, the witness, and a node re-run all see the same 16 points.
Solid orange = the max-margin boundary; dashed = the kerbs; ringed points = support vectors. Every click samples another valid separator — none is wider. The width is computed on the spot, never looked up.
What the machine produces, proven: the unique widest street — a normal w, an offset, a margin, and the handful of support vectors that pin it. Every one of the 16 points is classified correctly and lies at least a full margin from the line. Remove any non-support point and the answer is unchanged; move a support vector and it moves.
The blue team's witness (left) confirms this width is maximal live; the red team (right) tries to pass off a narrower street as the answer.
And "widest margin ⇒ best generalization" is a bound, not a theorem about any one dataset: the VC/margin guarantee can be loose, and under heavy label noise a narrower, softer boundary often tests better. Maximum margin is a principled default — not a promise.
"The SVM has no hyperparameters — the margin fixes everything." Cut. Soft-margin C and the kernel (and its parameters) are chosen, usually by cross-validation. Only the hard, separable case is parameter-free — and that is the toy case shown here.
"Maximum margin always generalizes best." Cut. It minimizes an upper bound on risk; the bound can be loose and noise can beat it. Kept honestly: it is a strong, well-motivated prior.
"More support vectors means a better fit." Kept, corrected. Fewer support vectors means a sparser, often better-generalizing model; the count is a complexity signal, not a quality score.
The red team's move: swap the solver for an arbitrary separator — the perpendicular bisector of the two class centroids. It still separates every point, so it looks valid — but its street is narrow. The blue team's witness (window 7) is watching.
Substitute a narrower separating line for the true maximum. The witness recomputes, finds a sampled line wider than the reported street, disagrees, and turns red. Nothing is faked; the attack is a real (worse) classifier and it is caught.