13 Research methods, statistics, critical review and evidence-based practice13.9 Differential and analytic statistics
13.9.7 Multiple testing and multiplicity
2 min read1 figure
Written by Resident Doctor Committee · Published
New in the 2027 syllabus
The College added this topic for exams from January 2027.
What the problem is
Every test carries some chance of a false positive. The more tests you run, the more likely one comes up "significant" by chance.
Multiplicity means there are several statistical chances to support the same claim. Sources include multiple outcomes, subgroups, treatment comparisons and interim analyses (looks at the data before the trial ends).
Family-wise error is the probability of at least one false rejection (wrongly rejecting a true null hypothesis) within a defined family of hypotheses.
False discovery rate is the expected proportion of false rejections among the hypotheses rejected. It counts as zero when none are rejected.
How quickly the risk grows
Assume every null hypothesis is true and each of five independent tests uses alpha 0.05.
Input | Value |
|---|---|
Number of tests | 5 |
Probability of no false positive per test | 0.95 |
The chance of no false positive across all five is 0.95^5 = 0.774.
So the chance of at least one false positive is 1 minus 0.774 = 0.226, about 23%.
If the tests are dependent, the exact figure changes. You still need to consider multiplicity.
Ways to control it
Method | How it works | Main point |
|---|---|---|
Bonferroni | Divide the family alpha by the number of tests | Controls family-wise error without needing independent tests |
Holm | Compare ordered p values with step-by-step thresholds | Controls family-wise error and is less conservative than simple Bonferroni |
Benjamini-Hochberg | Compare ordered p values with increasing thresholds | Controls false discovery rate under specified dependence conditions |
Hierarchical testing | Test outcomes in an order fixed in advance | A later confirmatory claim counts only if earlier tests succeeded |
Bonferroni worked example
Outcome | Unadjusted p value |
|---|---|
A | 0.008 |
B | 0.030 |
C | 0.200 |
D | 0.400 |
E | 0.900 |
Set the family alpha at 0.05 and divide by 5, giving 0.01 per test.
Only outcome A meets the corrected threshold.
Outcome B would pass at 0.05 alone, but here it does not support a family-wise confirmatory claim.
Appraising a paper
Prespecified families. The correction protects only the claims in a family defined in advance. Choosing a small family after seeing the results removes that protection.
Exploratory findings can generate hypotheses, if authors say they are exploratory and report the analyses they ran.
Repeated interim testing needs planned stopping boundaries or another valid sequential method. Testing at 0.05 at every look inflates false positives.
Correction can reduce power (the chance of detecting a real effect), so test a limited set of clinically meaningful outcomes rather than everything.