MOMUS · the exterior witness · the peer

THE NULL HYPOTHESIS p < 0.05 ?

'Significant at p<0.05' does not mean 95% likely to be true. Whether a positive result is real depends on how many of the hypotheses you test are true in the first place — the base rate. Test a field of mostly-false ideas and most of your 'significant' hits are false alarms, no matter how careful the p-value.

◆ LIT▲ AMBER
BASE RATE (true) POWER α
true hypotheses
20%
false 'positives'
true positives
false-discovery rate
P(true | significant)

◆ LIT — verified / checkable

Over a field of hypotheses, a fraction π are true. Testing at significance α with power (1−β) yields true positives = power·π and false positives = α·(1−π); the false-discovery rate among the 'significant' results is α(1−π)/(α(1−π)+power·π), and P(true | significant) is its complement — all exact, computed live. So p<0.05 gives 95% confidence only when the base rate is high; test mostly-false ideas and a majority of your discoveries are false, exactly as the readout shows.

▲ AMBER — the figure

A single-test framing (real research runs many correlated tests, and p-hacking makes effective α worse); π is a modelling assumption, not a measured quantity. The Bayesian point — significance without a base rate is not a truth probability — is exact and is the whole lesson.

MOMUS: no work is above critique — not even the critique.
David Lee Wise / ROOT0 / TriPod LLC  ·  with AVAN the peer