13 Research methods, statistics, critical review and evidence-based practice13.9 Differential and analytic statistics

13.9.2 Alpha levels and p values

2 min read1 figure

Written by Resident Doctor Committee · Published

Try it

Power, alpha and beta

Interactive

With 64 participants per group and an effect size of 0.50, the study has 81% power at an alpha of 0.05. The shaded tails are the false positive rate under the null and the false negative rate under the alternative.Significance cut-off-4-20246Observed difference, in standard errors

If there is no real differenceIf the real difference is this size

A standardised difference: 0.2 small, 0.5 moderate, 0.8 large.

Power
80.7%chance of finding a real effect
Type II error rate
19.3%missing a real effect
Type I error rate
5%calling noise a finding
Needed for 80% power
63 per group
Needed for 90% power
85 per group

Demanding a smaller alpha moves the cut-off to the right, which shrinks the false positive tail and grows the false negative one. Only more participants, or a bigger real effect, shrinks both.

Alpha is the shaded tail of the null curve, fixed before the study. Switch it from 0.05 to 0.01 and watch beta grow and power fall, then add participants to win the power back.

New in the 2027 syllabus

The College added this topic for exams from January 2027.

What alpha and p values are

Alpha is the significance threshold, chosen in advance and commonly 0.05. It is the long-run probability that your testing procedure rejects a true null hypothesis (a false positive).

The p value is the probability of a result at least as extreme as the one observed, assuming the null model and its assumptions hold.

Prespecification fixes the outcome, the analysis, the sidedness (one-sided or two-sided) and the significance threshold before anyone sees the result. This reduces selective choices.

Result at alpha 0.05

Decision

What is still uncertain

p = 0.03

Reject the null under this procedure

Effect size, clinical importance and risk of bias

p = 0.20

Do not reject the null

Whether an important effect was missed

What a p value does not tell you

  • A p value is not the probability that the null is true. Nor is it the probability that the finding occurred by chance alone.

  • A non-significant result does not show equivalence or absence of effect. The interval may still include large benefits and harms.

Choosing a threshold

  • Lower alpha gives fewer false-positive decisions. But with sample size and effect held constant, it usually lowers power (the chance of detecting a true effect) and raises the chance of a false negative.

  • A one-sided test puts the rejection region in one direction, chosen in advance. Choosing the direction after seeing the data invalidates its stated error rate.

  • Multiple outcomes or interim looks (analyses part-way through a study) need a strategy for multiplicity if a family of confirmatory claims is intended.

A mathematical illustration for a two-sided standard-normal test. Alpha sets the rejection region before seeing the data; the observed statistic sets the p-value tail area. Neither area is the probability that the null hypothesis is true. Original illustration drawn in Python for MRCPsychiatrist; see images/ATTRIBUTION.md

Interpreting results

  • Report effect estimates and confidence intervals with p values, so readers can judge size and precision.

  • Borderline values are not a sharp scientific divide. Under the same model, p = 0.049 and p = 0.051 give very similar evidence.

  • Check the model assumptions and look for complete reporting. A small p value cannot fix biased sampling or an inappropriate analysis.

Revision material for the MRCPsych examinations. Not clinical guidance, and not for use in patient care.