The do-operator for a neural network. A model runs on a clean input and a corrupted one; you reach into the corrupted run, overwrite one node's activation with the value it held on the clean run, recompute everything downstream, and watch the output move. How much of the clean answer comes back is the causal effect of that node. Down the center, data flows: activations go in, the patch intervenes, the recovered effect comes out. The blue team builds the graph and witnesses it; the red team tries to break the reading.
source Meng, Bau, Andonian, Belinkov, Locating and Editing Factual Associations in GPT (ROME), NeurIPS 2022 — arxiv.org/abs/2202.05262; causal-tracing lineage from Vig et al., Causal Mediation Analysis (2020). Rendered, not quoted.
Not a trained net — a hand-wired computational graph with known structure, so every readout has a closed form. Five nodes, one output:
The output depends only on n0 → n1 → n3 → out. Nodes n2, n4 compute, but nothing reads them. Live per-node recovered effect for the current inputs:
| node | on path? | recovered |
|---|
This is Pearl's do-operator aimed one level down — not at a variable in the world, but at an activation on the residual stream. Fixing a node's value and recomputing the rest is an intervention: it severs the node from its own causes and asks what its value alone forces downstream.
That makes it the workhorse of circuit-finding: sweep the patch across every node and layer, and the map of large effects localizes where a computation lives. Its neighbour is the-do-calculus — this sphere is that sphere's operator, applied to a network. Each sphere is the next one's premise.
The blue team's live check: re-run the patch over every node and confirm on-path nodes fully restore (recovered = 1) while off-path nodes have zero effect — and confirm the patch is a real recompute, not a read. If red tampers, this badge is where it shows.
Activation patching needs two forward passes. A clean input produces the answer we want to explain; a corrupted input (here, a different number) breaks it. Each pass leaves a full set of node activations:
| node | clean | corrupt |
|---|
The gap between the two clean/corrupt outputs is the budget the patch tries to recover. Pick a node, restore its clean value into the corrupt run — that is what you feed the panel below.
Full patch: node's clean value replaces its corrupt value, then all downstream nodes recompute.
Change any control — the recovered effect is computed by re-running the graph under the intervention, never looked up.
What the machine produces, proven: for this graph, patching an on-path node (n0, n1, n3) recovers exactly 1 — the full clean output — while an off-path node (n2, n4) recovers exactly 0. The effect map matches the graph's causal skeleton, to 1e-9. The current node's effect is above; the full map is the output.
The blue team's witness (left) confirms this map live; the red team (right) tries to make it lie.
The attribution also depends entirely on the corruption you chose: a different corrupt input lights up different nodes. Patching says where a change propagates, not what algorithm runs there — this toy graph is exact only because we built its wiring by hand.
"A high patching effect proves the node computes the fact." Cut. It proves counterfactual influence under one corruption — backup circuits and downstream interactions can inflate or hide it.
"Zero effect means the node is irrelevant." Cut. Only for this corruption; a redundant node can matter under a different one, or be masked by self-repair.
"Activation patching is just ablation." Kept, corrected. Ablation zeroes or mean-fills to destroy a value; patching replaces it with a specific clean value — a directed do(), not deletion.
The red team's move: swap the full patch for a half-patch — average the clean and corrupt values instead of replacing. Now the relevant node only partially restores the output. The blue team's witness (window 7) is watching.
Average instead of replace and n1 recovers 0.5, not 1 — the witness recomputes, sees the on-path node fail to fully restore, and turns red. Nothing is faked; the attack is real and it is caught.