From the outside of the box, decide one bit: was this sample in the training set? A model fits its own data too well — members sit at lower loss than strangers. Threshold the loss and you have an attack. Down the center, data flows: the losses go in, the attacker sweeps a threshold, the leak comes out as an AUC. The blue team builds the attack and proves it; the red team flips the rule to break it.
source Shokri, Stronati, Song & Shmatikov, Membership Inference Attacks Against Machine Learning Models (IEEE S&P 2017) — arxiv.org/abs/1610.05820. Rendered, not quoted.
The attacker sees only per-example loss (or confidence). Rule: predict member when loss < τ. The member-score is −loss, so low loss ⇒ high score. Sweep τ over every value and the whole ROC falls out.
The attack's power is one number, the area under that ROC — and it has a closed form. AUC equals the Mann–Whitney probability that a random member scores above a random stranger:
AUC = P( lossmember < lossstranger ) + ½·P( tie ) — rank-based, so it never changes under a monotone rescale of the score.
Exact ground truth on a constructed model — members {0,1,2,3}, strangers shifted by an integer gap g:
| gap g | AUC (closed form) | leak |
|---|
The model remembers who it studied. This sphere makes that memory measurable from outside the box; the next one builds the wall against it.
the-differential-privacy neighbour adds calibrated noise so that a single record's presence changes the output by a bounded ε — collapsing exactly the member/stranger gap this attack feeds on. Overfitting is the door; DP is the lock. Each sphere is the next one's premise.
The blue team's live check: recompute the closed-form AUCs over the constructed model and confirm them against the known truth — separated ⇒ AUC>½, overlap ⇒ AUC→½, rank-invariant. If red flips the rule, this badge is where it shows.
Two constructed piles of per-example loss, drawn with a fixed-seed PRNG: members (the model trained on them) and strangers (it did not). The only knob is the train–test gap — how much better the model fits its own data. That gap is the entire signal; the attacker never sees the membership labels.
A well-generalized model has a small gap: the piles overlap and there is almost nothing to read. An overfit model pulls the member pile down and left — and hands the attacker a lever.
A monotone rescale moves the histogram but not a single rank — the AUC is provably unchanged.
Every quantity is computed on the spot from the two loss piles — the AUC is the Mann–Whitney rank statistic, never a stored number.
What the machine produces, proven on the constructed model: a separated gap gives a loss-threshold AUC of 0.875 at g=2 and 0.969 at g=3; as the gap closes the AUC falls monotonically to exactly 0.5 — a perfectly generalized model leaks nothing. The number is invariant to any monotone rescale of the score.
The witness (left) reconfirms these closed forms live; the red team (right) flips the rule to try to make the leak lie.
And the honest limit cuts the other way: with a near-zero generalization gap there is no signal — AUC≈½ is not the attack failing, it is the model genuinely not leaking. A single global threshold also ignores that "easy" examples are low-loss whether member or not; per-example calibration is what fixes it.
"High test accuracy means the model is private." Cut. Accuracy and privacy are different axes; it is the gap between train and test loss that leaks — a model can be accurate and still memorize.
"The attacker needs the training data." Cut. It needs only query access to the loss/confidence for the target point. Membership is inferred, not looked up.
"AUC 0.5 means the attack is broken." Kept, corrected. 0.5 is the honest floor — no leak — not an error. Below 0.5 means the rule is inverted, which is exactly the tamper.
The red team's move: flip the decision rule — predict "member" when the loss is high. On separated data the attacker now scores below chance, and the "AUC>½ on a real gap" guarantee fails.
Flip it and the ROC bows under the diagonal: AUC drops below 0.5. The witness (window 7) recomputes, disagrees with the known 0.875, and turns red. Nothing is faked; the attack is real and it is caught.