A defense that hides the gradient looks robust and is not. Point a gradient attack (FGSM) at a classifier whose gradient has been flattened to zero and the attack collapses — the model appears safe. But the boundary never moved: a gradient-free attack that never touches a derivative still flips the class. The blue team builds the constructed model with closed-form margins; the red team shows that a passing FGSM is not evidence of robustness.
source Athalye, Carlini & Wagner, Obfuscated Gradients Give a False Sense of Security (ICML 2018) — arxiv.org/abs/1802.00420. Rendered, not quoted.
Everything is closed-form. The classifier is linear: f(x) = w·x with w = (3, −4), class = sign(f). The test point is x₀ = (1.5, 1.0), so s₀ = f(x₀) = +0.5 → class +1, a thin margin.
The “defense” wraps the logit in a quantizer q(s)=¼·round(4s). The staircase is piecewise-constant, so its gradient is exactly 0 almost everywhere — but sign(f) is untouched, so the decision boundary does not move.
Live closed-form quantities (recomputed, never looked up):
| quantity | closed form | value |
|---|
The neighbouring sphere the-fgsm builds the fast gradient sign attack: one step of ε·sign(∇). This sphere is its shadow: a defense that makes FGSM fail proves nothing.
Kill the gradient and FGSM starves — but the margin is unchanged, so a query-based attack walks straight through. That is why evaluations must be adaptive: attack the defense that is actually there, not the gradient it pretends to hide. This is the audit that broke a generation of defense papers. Each sphere is the next one’s premise.
The blue team’s live check: recompute both attacks from scratch and confirm the honest verdict — FGSM held (masked) yet the gradient-free attack flipped the class, so the model is not robust. If red judges by FGSM alone, this badge turns red.
Feed the engine three things: the model (w, x₀ above), a perturbation budget ε under the L∞ ball, and a choice of attack. Two attacks, one budget:
| attack | uses gradient? | closed-form optimum |
|---|---|---|
| FGSM | yes | δ = ε·sign(∇Loss) |
| grad-free | no | δ = −ε·sign(w) |
For a linear model the L∞-optimal step is −ε·sign(w) (shifts s by −ε·‖w‖₁) and the minimal L2 flip is the margin |s₀|/‖w‖₂. The gradient-free attacker recovers the L∞ optimum by query-only random search — no derivative required.
Mask on, FGSM: the quantizer flattens the gradient to zero, so the step is zero and the class holds — it looks robust.
Every number is computed on the spot from w, x₀ and ε — the masked gradient norm, the step, the shifted score, the flip. Nothing is looked up.
What the machine proves: under the mask, FGSM’s step is 0 and the class holds — yet at ε=0.1 the gradient-free attack drives s₀=+0.5 to −0.2 and the class flips. The masked gradient norm is 0 while the true ‖∇f‖ = ‖w‖₂ = 5. The boundary never moved; only the gradient lied.
The witness (left) confirms this live; the red team (right) tries to call it safe on the FGSM result alone.
Of the ICML-2018 defenses examined, the paper reports that 7 of 9 relied on obfuscated gradients and were circumvented. A tell: black-box attacks beating white-box, or unbounded ε failing to reach 0% accuracy — both mean the gradient, not the model, is doing the “defending.”
“FGSM accuracy stayed high, so the model is robust.” Cut. High FGSM accuracy under a masked gradient is the symptom of the flaw, not evidence against it — the gradient-free attack in the panel flips the same point.
“Non-differentiable = unattackable.” Cut. Non-differentiable only blocks the naive gradient; BPDA/query attacks route around it. Undifferentiable ≠ unbreakable.
“The defense raised the margin.” Corrected. Here the true margin |s₀|/‖w‖₂ = 0.1 is unchanged by masking — the engine asserts the boundary is byte-identical. Masking is a numerical artifact of the gradient, not a real margin.
The red team’s move: judge robustness by the FGSM result alone and ignore the gradient-free witness. FGSM failed — so declare the model safe.
Trust FGSM alone and the verdict flips to “SAFE” — but the model is exactly as broken as before. The witness (window 7) still runs the gradient-free attack, still sees the flip, and turns red. That gap is the false sense of security.