13 Research methods, statistics, critical review and evidence-based practice13.4 Systematic reviews
13.4.3 Heterogeneity and publication bias
6 min read3 figures
Written by Resident Doctor Committee · Published
Publication and language of publication bias
What publication bias is
Publication bias arises when a study's results influence whether, or when, it is published.
It arises mainly because investigators never submit studies with null or unwelcome results, the file drawer problem, rather than because journals reject them.
It is a between-study reporting bias acting on which studies are available, unlike within-study biases such as attrition or recall bias.
Outcome reporting bias, where only some pre-specified outcomes appear, and time-lag bias, where negative results are published later, distort the evidence even when studies are found.
Language bias and searching
A systematic review search must reach unpublished and non-English-language reports, across more than one database, to limit publication and language bias.
Restricting to English excludes a whole class of reports, and the excluded studies need not resemble those included in either direction.
Cochrane recommends searching without language restrictions where feasible.
Funnel plots
A funnel plot shows each study's effect size against its precision, usually the standard error, and is the graphical method for detecting publication bias in a meta-analysis.
Small studies scatter more widely than large ones, and without bias studies scatter symmetrically about the pooled effect.
An asymmetric funnel is the conventional visible sign of publication bias, and publication bias is its usual explanation.
The gap lies at the base, among small, imprecise, underpowered studies on the null or unfavourable side, because small negative trials go unpublished while large trials are published whatever they find.
Asymmetry can also come from heterogeneity, study quality or chance.
A funnel plot detects publication bias, a forest plot shows each study's effect and the pooled estimate, and I-squared measures heterogeneity.
Detecting and correcting for missing studies
Methods that look for absent trials (funnel plot, Egger's test, fail-safe N, registry comparison) assess publication bias. Statistics computed only from included trials, such as I-squared or the pooled confidence interval, cannot.
Method | What it does |
|---|---|
Egger's test | Gives a warning signal of asymmetry, not proof of bias; lacks power with fewer than about ten studies |
Rosenthal's fail-safe N | Counts the unpublished null studies, with zero effect, that would make a significant pooled result non-significant |
Duval and Tweedie's trim-and-fill | Imputes missing studies, assuming symmetrical scatter about the pooled effect, and so lowers an inflated pooled effect |
Registry comparison | Matches reports against registrations and protocols to find missing studies and dropped or changed outcomes |
Forest plots
On a forest plot a study is significant if its 95% confidence interval does not cross the line of no effect.
The pooled diamond is judged separately, so individual studies can be non-significant while the pooled estimate is significant.
Prevention through registration
Since 2005 the ICMJE has required prospective registration in a public registry as a condition of publication.
The AllTrials campaign, launched in 2013, asks for registration of older trials and posting of results whether or not a journal accepts the paper.
Registration records that a trial exists but does not guarantee access to its results.
Appraising a review
GRADE downgrades for publication bias, which lowers confidence in a result without reversing it.
When submitting a review of a drug, the interest that most needs declaring is a paid relationship with a company that stands to gain or lose from its conclusions, such as the maker of a competing product.
Recognising statistical heterogeneity
What heterogeneity is
Statistical heterogeneity is variation in study effects beyond what chance (sampling error) would produce.
Cochran's Q and I squared
Statistic | What it is | What it tells you |
|---|---|---|
Cochran's Q | A chi-squared test with k minus 1 degrees of freedom for k studies | Only whether heterogeneity is present, as a p value |
I squared | The percentage of variation between study results due to real between-study heterogeneity rather than chance | How much heterogeneity there is |
I squared = (Q minus degrees of freedom) / Q, as a percentage, with negative values set to zero; Q of 24.0 on 9 degrees of freedom gives 62.5%.
An I squared of 0% with a non-significant Q means no observed statistical heterogeneity beyond chance.
Q has low power with few or small studies, so a non-significant result does not rule heterogeneity out.
Cochrane's bands overlap and are guides, not fixed cut-offs.
I squared | Band |
|---|---|
0% to 40% | Low, might not be important |
30% to 60% | Moderate |
50% to 90% | Substantial (62% falls here) |
75% to 100% | Considerable |
Galbraith (radial) plots
Feature | Meaning |
|---|---|
Vertical axis | Standardised effect: the estimate divided by its standard error |
Horizontal axis | Precision: 1 divided by the standard error |
Slope of the fitted line through the origin | The pooled effect |
Bounds | Lines about 2 (1.96) vertical units either side of the fitted line |
A standardised effect near zero marks a study showing no effect.
Dividing the standardised effect by the precision returns the study's log odds ratio.
A study's own effect size is the gradient of the line from the origin to its point; its height reflects significance, not effect size.
Fixed-effect weight is the square of the precision, so the rightmost study carries the largest weight.
An outlier lies beyond the bounds and shows heterogeneity; Castellano 2017 sat about 3 units above the line.
About one study in twenty falls outside the bounds by chance, so outliers need investigating, not automatic exclusion.
Explaining and handling it
Subgroup analysis compares pooled effects across study categories with a formal interaction test; significance in one subgroup but not another is not enough.
Random-effects analysis estimates an average across varying effects but does not explain why they vary.
Sensitivity analysis in meta analysis
What a sensitivity analysis is
Sensitivity analysis repeats a meta-analysis under another defensible decision or assumption to test whether the conclusion holds.
Reviewers should prespecify analyses where possible, because choosing an alternative only for the answer it gives is selective reporting.
Subgroup analysis differs: it asks whether the effect differs between kinds of study or patient, and it needs a formal test for interaction.
Common checks
Check | What is varied |
|---|---|
Risk of bias | Exclude studies at high risk of bias |
Imputed statistics | Exclude studies with borrowed standard deviations, or rerun across plausible values |
Model | Fixed effect against random effects, which gives small studies more weight |
Leave-one-out | Omit each study in turn to find influential results |
Eligibility and time point | Age boundary, dose range, outcome time point |
Adjusting for publication bias
These methods explore assumptions. They cannot show what unpublished studies found.
Trim-and-fill corrects funnel plot asymmetry from publication bias by imputing the missing studies, which are small, imprecise studies with null or unfavourable results, on the side opposite the excess.
It removes the studies driving the asymmetry, re-estimates the centre, then adds mirrored hypothetical studies.
Its adjusted estimate depends on the assumed cause of asymmetry, so genuine heterogeneity can give a misleading adjustment.
Egger's test regresses the estimate divided by its standard error against 1 divided by the standard error, and an intercept away from zero suggests small-study effects rather than publication bias specifically.
Selection models, such as Copas models, test specified relationships between which studies are published and their results.
Reporting
Reviewers report every analysis, with the alternative assumption, the change in effect size and precision, and the effect on the conclusion.
Agreement shows robustness to the choices tested, not freedom from all bias.