◄ WORLD V · SONNY 5DART 225 · a helldive at the net

THE DIFFERENTIAL TEST two things that should agree, and the input where they don't

When two implementations are supposed to compute the same thing — two compilers, two JSON parsers, a fast path and a slow path, a new version and the old — feed them the same inputs and look for the one where they disagree. You need no oracle: the disagreement itself is the bug (at least one of them is wrong). Differential testing turns “is this correct?” into “do these agree?” — a question you can answer by search.

THE TECHNIQUE run two impls on the same inputs; the first divergence is a bug

Two implementations of a triangular sum — a reference recursion and a closed-form with a planted bug — are fed the same inputs. The checker returns the first input where they diverge: live demo







  
HISTORY & CREDIT McKeeman, 1998

  
“Without a spec of the correct answer, you cannot test.” — you can, if you have two things that must agree. Their disagreement needs no oracle: it proves at least one is wrong. This is how the hardest software — compilers — is actually tested. cited

  
1998 · William McKeeman — “Differential Testing for Software”: generate inputs, run multiple implementations, flag divergence.
2011 · Yang, Chen, Eide & Regehr — Csmith differentially tests C compilers, finding hundreds of GCC/Clang bugs.
now · parsers, crypto libraries, floating-point, and databases are all cross-checked implementation-against-implementation.

  
Two witnesses who must agree, and the case where they don't. No judge of truth is needed — only the contradiction, which convicts at least one of them. McKeeman 1998






  
RECOMMEND FOR I-13 first diverging input between two impls, computed

  
On the canonical compiler, a reference sum and a closed-form with a bug planted at n=7 are compared over 1..50; the checker returns the first divergence:

  
$ i13 run differential.i13   # sumA = recursion, sumB = n(n+1)/2 with a bug at 7
first_diff = 7        -- sumA(7)=28, sumB(7)=29: the two impls DISAGREE. one is wrong.

  
Recommend: differential testing is LIT for I-13 — verified a reference recursion and a closed-form triangular sum agree everywhere except the planted bug, and the checker returns the first diverging input n=7. “Is it correct?” becomes “do they agree?” — answerable by pure search, no oracle, the way real compilers are tested.