THE MUTATION TESTING who tests the tests? break the code on purpose and see if they notice
A green test suite proves nothing if the tests are toothless. Mutation testing audits the tests by sabotaging the program: it injects small deliberate faults — a + becomes a −, a < becomes a ≤ — producing mutants, and reruns the suite. A good test kills the mutant (fails on it). A mutant that survives is a bug the tests would never catch. The mutation score — mutants killed ÷ total — is how strong your tests really are.
THE TECHNIQUE inject faults; a test that fails KILLS the mutant, a survivor exposes a weak test
The same three mutants of add(a,b) face two tests. A strong test kills all three; a weak test lets one survive — the mutation score exposes exactly how toothless it is: live demo
HISTORY & CREDIT DeMillo, Lipton & Sayward, 1978
“All tests pass, so the code is well-tested.” — passing tests only prove the tests agree with the current code, not that they would catch a change. Mutation testing measures the thing coverage cannot: whether your tests would actually notice if the code were wrong. A high line-coverage suite can have a terrible mutation score. cited
1971–78 · Richard Lipton (student idea, 1971) then DeMillo, Lipton & Sayward (1978) — “Hints on Test Data Selection”: seed faults, measure which tests catch them. the coupling effect · tests that catch simple mutants tend to catch complex faults too — the empirical bet mutation testing rests on. now · PIT (Java), mutmut (Python), Stryker (JS) — mutation testing in CI, grading test suites.
It is the falsifier turned on the falsifiers: to trust a test that finds bugs, plant a bug and confirm the test finds that. A suite that survives no mutants is a suite that would survive your bugs too. DeMillo et al. 1978
RECOMMEND FOR I-13 mutation score: killed vs survived, computed
On the canonical compiler, three mutants of add (−, ×, &) meet two tests: a strong one kills all three, a weak one lets a mutant survive:
$ i13 run mutation.i13 # mutants of add: sub, mul, and
strong = 3 -- test add(3,4)==7 kills all 3 mutants (-1, 12, 0 all != 7): adequate
weak = 2 -- test add(2,2)==4 kills 2, but mul (2*2=4) SURVIVES: a weak test exposed
Recommend: mutation testing is LIT for I-13 — verified that the test add(3,4)==7 kills all 3 mutants (score 3/3, adequate) while add(2,2)==4 kills only 2 — the × mutant survives because 2×2==4. The falsifier aimed at the tests themselves: plant a bug, and demand the suite catch it.