A hand-built linear readout over two inputs x+, x- and two components. Every weight is fixed; every readout has a closed form.
y(x) = wC·aC(x) + wU·aU(x) + b
aC: [x+ → 2, x- → 0] wC=2
aU: [x+ → 0.5, x- → 0.5] wU=1 b=0
C is the key: it varies with the class, so it carries the split. U is constant across classes — pure offset, no class signal. Target metric = margin = y(x+)−y(x-). Intact margin = 4.
The necessity test — the destructive twin of the-activation-patching. Patching writes a clean activation in and asks "is this enough?" (sufficiency). Ablation takes a component out and asks "does it break?" (necessity). Same circuit, opposite direction: one adds signal to find what suffices, one removes signal to find what is load-bearing.
Re-runs the necessity property against the current engine state: mean-ablating C must drive margin to baseline 0. Trip the tamper in window 6 and this flips red.
witness idle
Two probes, one per class, and a choice of what to knock out and how.
inputs = { x+ , x- } · components = { C=key , U=unrelated }
ablation ∈ { zero (→0) , mean (→E[a]) }
| row | y(x+) | y(x-) | margin | behavior |
|---|---|---|---|---|
| intact | ||||
| ablated |
booting…
Proven at boot: knock out C (zero or mean) → margin 0, the split is GONE. Knock out U → margin stays 4, unchanged. Zero vs mean on C agree on margin (both 0) but leave different levels, differing by exactly wC·E[aC] = 2.
"A drop after ablation only proves the component participates, not that it is the mechanism. You broke the network's operating point — every downstream layer now sees off-distribution garbage. That is knock-on damage, not localization."
True on real nets: ablation is invasive. Off-distribution activations, compensating heads, and dead-neuron re-routing all muddy the read. This toy is linear and self-contained so the confound is absent here — that boundary is the honest claim, not a general one.
"If ablating a component doesn't change the output, the component is useless."
→ It may be redundant, backed-up, or its effect may cancel at this operating point. Ablation shows this path is not solely necessary — not that the feature is unused.
"Mean-ablation is a gentler zero-ablation."
→ They answer different questions. Zero removes the whole activation; mean removes only the variation and keeps the average/bias. On C both kill the margin, but leave different output levels (Δ = 2).
"Bigger drop = more important."
→ Magnitude conflates the metric's scale with mechanism. Necessity is a structural claim (does the behavior survive?), read against a baseline — window 8. AMBER on trained nets the baseline itself is a modeling choice.
Swap the honest global mean for a leaky class-conditional "mean" that returns each class's own average — i.e. the original activations. The removal removes nothing; the behavior survives; C now looks unnecessary.
state: honest. The witness (7) confirms necessity holds.