13 Research methods, statistics, critical review and evidence-based practice13.4 Systematic reviews
13.4.4 Role and limitations of meta analysis
4 min read2 figures
Written by Resident Doctor Committee · Published
What pooling buys
Meta-analysis combines estimates from several studies into one summary effect with a confidence interval. A systematic review finds and appraises the evidence and need not include one.
Single studies are often too small to give a precise, generalisable answer, so pooling raises the total sample size and reduces random error.
Pooling carries systematic error into the summary: bias, confounding and measurement error remain, so twenty biased trials give a precise but misleading estimate.
Similar studies of the same comparison are pooled and weighted by precision. They are not discarded, selected or vote-counted by whether each reached significance.
Randomised controlled trials are the design pooled most often, and the result is only as good as its inputs: garbage in, garbage out.
Types of meta-analysis
Pooling published summary results from trials that all compare one drug with placebo is a literature-based, standard direct (pairwise) meta-analysis. It is distinct from individual patient data, indirect network and cumulative meta-analysis.
A network meta-analysis estimates the effect between two treatments never compared head to head by linking them through shared comparators.
On a network diagram, a treatment (node) with no line to placebo has indirect evidence only.
Weighting and effect measures
Under inverse-variance weighting, weight = 1 / (standard error squared), so halving a study's standard error quadruples its weight.
Diagnostic accuracy statistics describe a test in one group and depend on prevalence, so they are not effect sizes pooled in a meta-analysis of treatment trials.
Reading a forest plot
Element | What it shows |
|---|---|
Square's position | The study's point estimate |
Square's area | The study's weight, not the size of its effect |
Horizontal line | The study's confidence interval, usually 95% |
Vertical line | No effect: 1 for ratios, 0 for mean differences and risk differences |
Diamond | The pooled estimate (centre) and its confidence interval (width) |
A study's weight, and so its square, grows with precision, mainly its sample size and number of events.
A ratio whose 95% confidence interval lies wholly on one side of 1 is significant, and the side gives the direction.
An interval crossing 1 supports no direction, whatever the point estimate.
Hazard ratios against a reference category
A reference category is fixed at exactly 1.00 by definition, so it has no interval and its square has no horizontal line.
Clozapine's adjusted hazard ratio of 0.74 (0.60 to 0.91) lies wholly below 1, so its lower risk is significant.
Few events widen an interval: quetiapine, with the fewest deaths (89) and person-years (5360), had the longest interval of any significant result (1.09 to 1.82).
An estimate that moves from just below 1 in the crude analysis to just above 1 after adjustment, with both intervals including 1, supports neither direction.
Antidepressants against placebo
A 2018 network meta-analysis pooled 522 double-blind trials of 21 antidepressants in 116,477 participants.
Plot | Odds ratio above 1 means | Best | Worst |
|---|---|---|---|
Efficacy | Greater odds of response on the drug | Amitriptyline 2.13, then mirtazapine 1.89 | Reboxetine 1.37 (lower limit 1.16), still better than placebo |
Acceptability | More people stopped the drug than placebo | Agomelatine 0.84 (0.72 to 0.97) | Clomipramine 1.30 (lower limit 1.01), worse than placebo |
Deciding whether to pool
Heterogeneity | Meaning |
|---|---|
Clinical | Different populations, treatments, comparators, outcomes or settings |
Methodological | Differences in design, conduct or analysis, such as blinding |
Statistical | More variation in effects than sampling error explains, measured by I squared |
Pooling different designs, such as randomised and uncontrolled studies, weakens the summary, so each design gets a separate synthesis.
Studies that fall into two groups with categorically different methods become two separate meta-analyses, not one forced pooled estimate.
Limitations pooling cannot repair
Publication bias makes published studies unrepresentative: of trials of 12 antidepressants registered with the US Food and Drug Administration, 31% were unpublished, inflating the apparent effect size by 32%.
A funnel plot shows study effects against precision. Asymmetry can reflect missing studies, true heterogeneity, poorer small trials or chance, so it cannot prove publication bias, and symmetry cannot exclude it.
Small-study effects are systematic differences between smaller and larger studies; random-effects models give small studies more weight, which can increase bias.
False precision arises when duplicate publications of one trial, a shared control group or several outcomes from one study are counted as independent.
The ecological fallacy is inferring patient-level relationships from associations between study averages, such as concluding that more severely ill individuals benefit more.