13.4.7 Appraising systematic reviews and meta analyses

Types of review

• A systematic review is secondary research: its units of observation are completed studies, not patients, and its eligibility criteria select studies.

• A review condenses a large literature into a manageable summary, whereas a randomised controlled trial generates new data.

• Gene Glass coined the term meta-analysis in 1976.

The core stages of a systematic review are:

1. Specify the question and the evidence sought.

2. Find the evidence.

3. Appraise it.

4. Synthesise it.

Randomisation is a step of primary trials, not of a review.

The core steps of a meta-analysis are:

1. Pre-specify inclusion and exclusion criteria.

2. Search the literature exhaustively.

3. Extract data from each included study.

4. Pool the results statistically.

The question, protocol and search

• The main objective of a review is read from its population, intervention, comparator and primary outcome, not from secondary outcomes or features it never measured.

• Eligibility criteria should be fixed before results can influence selection, and prospective registration, as on PROSPERO, helps reveal later changes.

• The search should be reproducible, with dates and restrictions reported, and should reach sources beyond journals.

• Several reports may describe one study, so reviewers match registration numbers, sites and sample sizes to avoid counting participants twice.

Missing studies

• Publication bias occurs when negative trials go unpublished, so a review built from journals alone misses them.

• Selective outcome reporting is different: negative trials are published, but with a positive slant.

• Including unpublished trials is the specific strength of a review that counters publication bias.

• Funnel-plot asymmetry tests generally need at least about 10 studies, and asymmetry can reflect publication bias, methodological differences or real heterogeneity.

Tap to enlarge

Risk of bias in included studies

• Summary quality scores were abandoned because their arbitrary weighting hides important flaws, so risk of bias is judged domain by domain.

• RoB 2 rates results from randomised trials as low risk, some concerns or high risk; one high-risk domain makes the overall judgement high risk.

• ROBINS-I rates results from non-randomised studies of interventions as low, moderate, serious or critical risk.

Reporting and appraisal frameworks

The design of the report decides the guideline, not the design of the studies it pools.

• PRISMA has a checklist and a flow diagram that tracks records from identification to inclusion.

• Reporting checklists certify transparent reporting, not low bias; AMSTAR 2 appraises the methods of a review itself.

• GRADE rates the certainty (quality) of a body of evidence across all studies reporting an outcome, and is used in systematic reviews and guideline development.

• GRADE downgrades for risk of bias, inconsistency, indirectness, imprecision and publication bias.

• The size of the pooled effect is the result of a meta-analysis, not a marker of its quality.

Tap to enlarge

Effect measures and pooling

• The standardised mean difference (Cohen's d, Hedges' g) is the difference in group means divided by the standard deviation of the scores, either pooled or of the control group.

• It carries no units, so studies using different outcome scales can be pooled.

• For a risk ratio, the no-difference value is exactly 1, and a value below 1 means lower risk in the numerator group.

• Studies are weighted by the precision of their estimates, largely sample size and event count, not by the size, direction or significance of their result.

• Significance is not clinical importance: the confidence interval shows significance, and the size of the estimate shows whether the effect matters clinically.

• Vote counting, which tallies significant studies, discards effect size and precision; five trials each non-significant at risk ratio 0.75 give a significant pooled interval of 0.64 to 0.88.

Reading a forest plot

The forest plot, informally a blobbogram, shows each study's effect estimate with its confidence interval, and the pooled estimate as a diamond.

Tap to enlarge

• A study is statistically significant only when its whole 95% confidence interval lies on one side of the line of no effect (1 for a ratio measure).

• An interval that touches or crosses the line of no effect is not significant.

• Funnel, Galbraith and L'Abbe plots examine bias or heterogeneity rather than displaying each study's effect with a pooled diamond.

• In a network meta-analysis plot, the thickness of the line joining two treatment nodes reflects the amount of direct head-to-head evidence, usually the number of trials comparing that pair.

Tap to enlarge

Heterogeneity

Heterogeneity is variation between studies, so a heterogeneity statistic needs at least two study estimates and has no meaning for a single trial.

• Statistical heterogeneity is quantified by Cochran's Q, I-squared and tau squared, and shown graphically by forest, Galbraith and L'Abbe plots.

• It cannot be read from interval width, number of studies, effect size or significance, and review steps aimed at other problems do not measure it.

• A significant Cochran's Q rejects the null hypothesis that all between-study variation is due to chance; it says nothing about the pooled effect, study quality or publication bias.

• On a forest plot, point estimates scattered more widely than their intervals allow, with poorly overlapping intervals, suggest heterogeneity.

• Visual inspection can suggest heterogeneity or its absence but cannot establish significance or prove homogeneity, which need Cochran's Q or I-squared.

• A random-effects model estimates an average across varying effects; it does not explain heterogeneity or make incompatible studies comparable.

• Subgroup analyses and meta-regression are unreliable with few studies, and significance in one subgroup but not another is not enough to believe a subgroup effect.