13 Research methods, statistics, critical review and evidence-based practice13.9 Differential and analytic statistics

13.9.1 Hypothesis testing

Creator: Resident Doctor Committee
Published: 2026-10-14T21:05:44.632Z

3 min read2 figures

Written by Resident Doctor Committee · Published 14 October 2026

Try it

Normal curve

Interactive

Lower bound (z)

Upper bound (z)

Mean

Only the axis labels move: the curve is drawn in z.

Standard deviation

Shaded area: 95.0%The exact 95% interval.
Outside the shading: 5.0%Split between the two tails, which need not be equal.
Lower bound: -2z = −1.96
Upper bound: 2z = +1.96

A z-score says how many standard deviations a value sits from the mean, so one curve serves every scale and the shaded fraction depends on z alone. That is why the 95% interval is at ±1.96 standard deviations whether the measure is an IQ, a blood level or a symptom score.

The shaded middle holds 95%, leaving 2.5% in each tail beyond 1.96 standard deviations. Set the lower limit to -4 and the upper to 1.645: the 5% now sits in one tail, so a one-tailed test needs only 1.645.

Null and alternative hypotheses

The null hypothesis (H0) states that there is no difference or association, such as a mean difference of zero or a risk ratio of one. It is what the test tries to reject.
The alternative hypothesis (H1), also called the research hypothesis, states that a real effect, difference or association exists. It is what the study seeks to support.

Alternative	What it predicts	When justified
Two-tailed (non-directional)	A change in either direction	Differences either way matter; the usual default
One-tailed (directional)	The direction of the effect	Direction fixed in advance, and an effect the other way is implausible or of no consequence

The p value

The p value is the probability of a result at least as extreme as the one observed, calculated from the data on the assumption that the null hypothesis is true.
A smaller p value means the data are less compatible with the null, which is stronger evidence against it.
A p value depends on both effect size and sample size, so it does not measure how large the effect is. Equal p values from trials of different size do not mean equal effects.
The p value is not the probability that the alternative hypothesis is true, it is not power, and it is not a measure of clinical importance.

The p value is not	Why
The probability that the null is true	The calculation assumes the null is true
The probability that the result happened by chance	It already assumes chance is the only explanation
One minus the probability that the alternative is true	The alternative never enters the calculation

Significance level, errors and power

The alpha (significance) level is the Type I error threshold, commonly 0.05, fixed in advance. If the p value falls below alpha, the null is rejected.
Significance is decided by comparing the p value with alpha, not by the size of the observed effect.
Alpha caps the long-run rate of rejecting a true null hypothesis.
A p value just above 0.05 is a non-significant trend: it is not a significant result and not proof of no effect.

Decision	Null true	Null false
Reject null	Type I error (false positive), probability alpha	Correct
Retain null	Correct	Type II error (false negative), probability beta

Two overlapping bell curves split by a vertical critical value line. The null curve's tail beyond the line is shaded red as alpha, the alternative curve's area before the line amber as beta, and the rest of the alternative curve green as power. — The red tail of the null curve beyond the critical value is alpha, the Type I error (false positive), the amber part of the alternative curve short of it is beta, the Type II error (false negative), and the green remainder is power, 1 minus beta; drawn one-sided, whereas a two-sided test shares alpha between both tails. Drawn for MRCPsychiatrist

Power is the probability of detecting a true effect, equal to 1 minus beta. It rises with sample size and with effect size.
Running many tests raises the chance of at least one false positive, and the Bonferroni correction divides alpha by the number of tests.

Statistical against clinical significance

Statistical and clinical significance are separate. Clinical importance is judged by reading the confidence interval (CI) against the minimal clinically important difference.
Failing to reject the null does not show that there is no effect or that treatments are equivalent.

Study	Result	Reading
Blood pressure, 50,000 per arm	0.4 mmHg difference, p = 0.001	Significant, but patient benefit is not shown
Weight, small study	3 kg difference, 95% CI -2 to +8 kg	Not significant, yet compatible with no difference and with an important effect

Four mean differences, labelled A to D, each drawn as a square with a horizontal confidence interval line, plotted against a grey zero line and dashed margins at minus 0.5 and plus 0.5. — Read each confidence interval against zero and the dashed margins: interval D crosses zero, so it is not significant, yet it also reaches past a margin, so it shows neither no difference nor equivalence, while interval C excludes zero but stays inside the margins. Lakens, CC BY-SA 4.0, via Wikimedia Commons

Revision material for the MRCPsych examinations. Not clinical guidance, and not for use in patient care.