Read a feature off the activations — with the control that keeps you honest. A linear probe is trained to predict a feature from hidden states; if it succeeds, the feature is said to be there. But a probe with enough capacity can predict anything — even a column of coin-flips. The fix (Hewitt & Liang, 2019) is a control task: train an identical probe on random labels; selectivity = real accuracy − control accuracy is the honest number. Down the center: activations go in, two probes fit, selectivity comes out. Blue builds it; red drops the control and calls a coin-flip a discovery.
source Alain & Bengio, Understanding intermediate layers using linear classifier probes (2016) — arxiv.org/abs/1610.01644 · Hewitt & Liang, Designing and Interpreting Probes with Control Tasks (2019) — arxiv.org/abs/1909.03368. Rendered, not quoted.
The activations here are constructed by hand, not trained: K feature-types, each a fixed random point in 48-dim space; the probe sees only the first d coordinates. The real label is a closed form of the hidden state (a sign, or an XOR of two signs). Two probes are fit by ridge least-squares (convex, closed-form — the global optimum, no training luck):
real probe → predicts the true feature. control probe → predicts a random label pinned per type (Hewitt–Liang's control task). Both report held-out accuracy:
| scenario | d | K | probe | control |
|---|
Selectivity is the gap. A real feature: big gap. Over-capacity: no gap.
The linear representation claims a feature lives along a direction in activation space. The probe is how you'd test that claim — fit a line, read the accuracy.
But a bare probe can't tell "the model represents it" from "the probe just learned it." Hewitt–Liang's control task is the cross-examination: subtract what the probe can learn from noise. This sphere puts the previous one on trial — each sphere is the next one's premise.
Blue's live check: recompute all three scenarios and confirm the honest verdicts — the linear feature reads REAL, the over-capacity case reads spurious (control high). If red drops the control, the coin-flip is called a discovery and this badge goes red.
Three regimes of hidden state feed the panel. Each is a set of feature-types; test tokens sit near their type's prototype:
| regime | the feature is… | probe sees |
|---|---|---|
| linear | on one direction | d=2, K=40 |
| nonlinear | XOR of two signs | d=6, K=40 |
| over‑capacity | on one direction | d=20, K=16 |
The only knobs are which regime and whether you subtract the control. Everything downstream is computed, never looked up.
Honest: report selectivity = probe − control. A high-capacity probe fitting random labels is subtracted away.
Least-squares is closed-form: same activations → same weights, every load. No seed roulette.
Switch to RAW ACC and the over-capacity case suddenly "discovers" a feature that is not there — that is the tamper, live.
What the machine proves, live: on a linearly-encoded feature the probe hits 100% and the control sits at ~55% — selectivity +0.45, a real find. On the over-capacity feature both probes hit 100% — selectivity 0.00: the probe fit random labels just as well, so its success means nothing. On XOR the linear probe stalls near chance — the feature is present but not linearly readable.
The witness (left) confirms these numbers live; the red team (right) tries to erase the control.
Selectivity fixes the "probe just memorized it" failure — not the "correlation isn't causation" failure. That needs interventions (causal mediation, activation patching), which no probe performs. A probe is a witness, not a surgeon.
"The probe hit 95%, so the model represents the feature." Cut. Without a control, 95% may be the probe's own capacity. Report selectivity — probe minus control — or report nothing.
"A stronger probe (deep MLP) is a better probe." Cut. More capacity lifts the control too; Hewitt–Liang show selectivity often drops. Favor the simplest probe that separates the task from its control.
"Low linear-probe accuracy means the feature is absent." Kept, corrected. It means not linearly present — see XOR: the probe reads chance while the feature is fully determined.
Red's move: drop the control. Report raw probe accuracy and call it selectivity. Now the over-capacity probe — fitting pure noise at 100% — looks like a landmark discovery. The witness (window 7) is watching.
Drop the control and raw accuracy stands in for selectivity: the over-capacity coin-flip reads REAL, the witness recomputes, disagrees with the honest verdict, and turns red. Nothing is faked; the attack is real and it is caught.