13 Research methods, statistics, critical review and evidence-based practice13.8 Descriptive statistics

13.8.8 Confidence intervals compared with p values

2 min read1 figure

Written by Resident Doctor Committee · Published

Try it

What a confidence interval tells you

Interactive

An estimate of 2.00 with a 95% confidence interval from -0.48 to 4.48, which includes the null value of 0.No difference2.00 (-0.48 to 4.48)-10-50510Difference between the groups

The only one of these you can usually do anything about.

Standard error
1.265
Margin of error
2.4791.96 standard errors
Interval
-0.48 to 4.48
Crosses the null?
Yesno difference is still compatible

To halve this interval you would need about 161 patients, four times as many. A wide interval that crosses the null has not shown that the treatment does nothing; it has shown that the study was too small to say.

The mean improvement stays at 2 points. Slide n from 40 to about 70 to see the interval stop crossing 0, push it up to 1000 for a narrow interval around the same small effect, then switch confidence to 99%.

What each number tells you

  • A p value measures how extreme the data are under the null model. It is not the probability that the null is true or that chance caused the result.

  • A confidence interval (CI) adds the size, direction and uncertainty of the effect.

  • 95% coverage refers to intervals repeated under the method's assumptions, not a 95% chance that the true value lies in this one interval.

  • The limits describe uncertainty about the effect. They are not proportions of patients or a reference range for individuals.

Reading significance from the interval

  • A 95% CI that excludes the null value corresponds to p below 0.05 in a two-sided test from the same model.

  • The null value is zero for a difference, including a standardised mean difference, and one for a ratio.

  • A study whose CI contains the null is not significant in either direction, however far its point estimate sits from the null and whatever its weight.

  • On a forest plot, a pooled diamond lying wholly on one side of the line of no effect is significant.

Sample size and precision

Precision rises with sample size, so a very narrow CI around a small difference with a very small p value points to a large sample. Very large trials make trivial differences significant.

Feature

Trial of 4,000

Trial of 40

Mean difference on the Montgomery-Åsberg Depression Rating Scale

1.0 point

2.0 points

95% CI

0.4 to 1.6

4.2 worse to 8.2 better

p value

0.001

0.53

  • Against a five-point important difference, the large trial is significant but clinically small.

  • A CI that contains the null and spans both worthwhile benefit and meaningful harm leaves the decision unresolved: it is inconclusive, not evidence of no effect.

Margins

Trial question

What the CI must do

Superiority

Exclude the null on the favourable side

Non-inferiority

Exclude the prespecified margin on the unfavourable side

Equivalence

Lie wholly between minus delta and plus delta

Harm

Keep its upper limit below an acceptable ceiling

Compare each interval with the grey null line at zero and the dashed margins at minus and plus 0.5: C excludes the null but lies wholly within the margins, so it is both significant and equivalent, while D crosses both the null and a margin and is neither. Lakens, CC BY-SA 4.0, via Wikimedia Commons

What neither number corrects

  • Neither corrects selection, measurement, attrition or reporting bias, and a larger sample makes a biased estimate more precise.

  • Multiplicity: twenty independent tests of true nulls at alpha 0.05 give one false positive on average and about a 64% chance of at least one.

Revision material for the MRCPsych examinations. Not clinical guidance, and not for use in patient care.