13 Research methods, statistics, critical review and evidence-based practice13.9 Differential and analytic statistics

13.9.1.1 Frequentist and Bayesian hypothesis testing

2 min read1 figure

Written by Resident Doctor Committee · Published

Try it

Updating a belief with a trial

Interactive

Prior, trial result and posteriorThe prior centres on 0.0 points, the trial found 3.0 points, and the posterior centres on 1.7 points, with a 89% chance that the true effect is above zero.-8-404812Difference in depression score (points)
Prior: belief before the trialLikelihood: what the trial alone saysPosterior: belief after the trial (shaded above zero)
Posterior effect
1.7 points95% credible interval -0.9 to 4.3. The data carry 56% of the weight.
Chance the treatment helps at all
89%
Chance it beats 2.0 points
40%The smallest difference worth having.
Bayes factor, effect against none
1.5Inconclusive
Estimate and 95% confidence interval
3.0 points-0.5 to 6.5
p value against no difference
0.094Not significant at 0.05.

The two readings answer different questions. The p value is the chance of data at least this extreme if the treatment did nothing, and the confidence interval is a range built by a method that captures the true effect in 95% of repeated trials. The posterior is a probability about the effect itself, given this trial and the prior. With a vague prior the credible interval and the confidence interval almost coincide; a sceptical prior pulls a small trial's result towards zero, and a large trial overrides any reasonable prior. The outcome standard deviation is 8 points in each arm.

A trial of 40 patients per arm finds a 3-point benefit, p = 0.094. Try the sceptical, vague and enthusiastic priors and watch the posterior, its credible interval and the Bayes factor move; then raise the patients per arm until the prior hardly matters.

New in the 2027 syllabus

The College added this topic for exams from January 2027.

Two approaches to uncertainty

Both approaches use data to learn about an unknown parameter, such as the true effect of a treatment.

Feature

Frequentist approach

Bayesian approach

Parameter

Fixed but unknown

Uncertainty about it is shown as a probability distribution

Data

Would vary if the study were repeated

Observed data update prior beliefs

Main inputs

Sampling model and analysis procedure

Prior distribution and likelihood

Typical interval

Confidence interval

Credible interval

Frequentist error rates describe how a procedure performs over repeated datasets under stated conditions. They are not the probability that a particular hypothesis is true.

The Bayesian approach has three parts:

  • Prior distribution: uncertainty about the parameter before the current data are added. It can reflect earlier evidence or a deliberately weak assumption.

  • Likelihood: how compatible the observed data are with different parameter values, under the model.

  • Posterior distribution: the prior and likelihood combined using Bayes' theorem. It describes uncertainty after seeing the data.

Comparing hypotheses

  • Bayes factor: compares how probable the observed data are under two specified models, averaging over each model's parameter uncertainty where relevant.

  • Posterior odds: prior odds multiplied by the Bayes factor. The Bayes factor alone is not the posterior probability of either hypothesis.

  • p value: a tail probability calculated under a null model. It cannot be turned into one unique Bayes factor without further modelling assumptions.

Worked example

Take two mutually exclusive hypotheses, H1 and H0.

Input

Value

Prior probability of H1

0.20

Prior probability of H0

0.80

Bayes factor favouring H1 over H0

6

  1. Turn probability into odds. Prior odds for H1 = 0.20 / 0.80 = 0.25.

  2. Multiply by the Bayes factor. Posterior odds = 0.25 x 6 = 1.5.

  3. Turn odds back into probability. Posterior probability of H1 = 1.5 / (1 + 1.5) = 0.60.

This answer holds only given the two models and the prior.

The chapter's two-hypothesis example. A Bayes factor multiplies prior odds, not prior probability; the posterior remains conditional on the hypotheses, model and prior. Original illustration drawn in Python for MRCPsychiatrist; see images/ATTRIBUTION.md

Intervals and appraisal

  • 95% confidence interval: a statement about the procedure. If its assumptions hold, the procedure covers the fixed true parameter in 95% of repeated applications.

  • 95% credible interval: contains 95% of the posterior probability for the parameter, given the model, prior and data.

  • Prior sensitivity: check whether other reasonable priors would materially change the conclusion.

  • Model checking: still needed in both approaches. Neither framework automatically removes bias, confounding or selective reporting.

Revision material for the MRCPsych examinations. Not clinical guidance, and not for use in patient care.