13 Research methods, statistics, critical review and evidence-based practice13.1 Introductory concepts

13.1.2 Sources and hierarchy of evidence

5 min read3 figures

Written by Resident Doctor Committee · Published

Sources of evidence

Primary, secondary and tertiary evidence

These tiers describe how far evidence has been processed, not how good it is.

Tier

What it is

Examples

Primary

Original data collected to answer one question

Trial report, cohort study, interview study

Secondary

A synthesis of primary studies

Systematic review, meta-analysis

Tertiary

Evidence digested for direct clinical use

Textbook, national guideline, British National Formulary

Filtered and unfiltered sources

  • Filtered (pre-appraised) resources, such as the Cochrane Database of Systematic Reviews, have already selected, appraised and synthesised the evidence.

  • Unfiltered sources, such as bibliographic indexes and trial registers, list studies without appraising them.

  • Databases differ in what they cover, so the first search depends on the kind of evidence needed.

Bibliographic databases

Database

What it covers

MEDLINE

The largest component of PubMed but not the whole of it; worldwide and multilingual

PsycINFO (APA PsycInfo)

Psychology and behavioural science journals, books, chapters and dissertations, with records back to the seventeenth century

CINAHL (Cumulative Index to Nursing and Allied Health Literature)

Nursing and allied health journals, including qualitative work on patient experience and practice-focused titles that MEDLINE misses

OpenGrey

Grey literature only

  • Grey literature is reports, theses, working papers and policy documents issued outside the usual commercial and academic publishers.

The Cochrane Library

The Cochrane Library does not duplicate MEDLINE, and it is neither exhaustive nor static.

Collection

What it holds

Cochrane Database of Systematic Reviews (CDSR)

Full-text, protocol-based systematic reviews of the effects of healthcare interventions, meant to be updated

Cochrane Central Register of Controlled Trials (CENTRAL)

A curated register of controlled trial reports only, built from bibliographic databases, handsearching and trial registries, so it lists ongoing and unpublished trials

  • DARE (Database of Abstracts of Reviews of Effects) held only structured abstracts and appraisals of reviews published elsewhere.

Trial registries and unpublished evidence

  • ClinicalTrials.gov is a trial registry that finds trials registered before recruitment but never written up, with their planned outcomes, completion dates and, for many, posted results.

  • Regulatory documents can hold unpublished outcomes: in a 2010 reboxetine review, 74% of patient data were unpublished, and the published reports had overstated benefit and understated harm.

  • Searching beyond journals, including grey literature and trial registers, reduces publication bias in a review.

An illustrative funnel plot of trial effect against standard error: precise trials cluster near the pooled effect (vertical line), while every small trial sits to the right, leaving empty the lower left where small unfavourable trials would fall. Registries and regulatory documents are where such missing trials are found. Illustrative data. Drawn for the MRCPsychiatrist statistics course

Matching the source to the question

Question

First source

Does this treatment work?

A systematic review of randomised controlled trials, or a single trial if none exists

What is the recommended management now?

A current national guideline

Is this a rare or long-term harm?

Large cohort data, pharmacovigilance databases and regulatory safety reports

How do patients experience this?

A qualitative or mixed methods study

Sample size and precision

  • Sample size governs precision (random error): a larger sample gives a smaller standard error and a narrower confidence interval.

  • It does not govern representativeness (systematic error) or control of confounding, which randomisation addresses.

As sample size rises from 16 to 64, the sampling distribution of the mean narrows around the true value, showing that a larger sample gives a smaller standard error and so a more precise estimate. Biggerj1, CC BY 4.0, via Wikimedia Commons

WHO World Mental Health Survey Initiative

  • The World Health Organization (WHO) World Mental Health Survey Initiative uses the fully structured WMH-CIDI (World Mental Health Composite International Diagnostic Interview).

  • Trained lay interviewers give it face to face to adult household samples in a self-selected subset of countries across income levels and regions.

  • England has no survey in the consortium, because its national psychiatric morbidity surveys use their own instrument.

Hierarchies of evidence for different questions

Primary and secondary evidence

  • Primary evidence is original research that collects new data from participants.

  • Secondary sources synthesise, appraise or interpret research already done: systematic reviews, meta-analyses, guidelines and reference texts.

The hierarchy depends on the question

The familiar pyramid ranks designs for treatment effects. Other questions have a different best design.

The familiar pyramid ranks designs for a treatment question: systematic reviews and meta-analyses sit above randomised controlled trials, then cohort and case-control studies, with case reports near the base, and the brackets separate primary studies from secondary sources. CFCF, CC BY-SA 4.0, via Wikimedia Commons

Question

Design with least inherent bias

Treatment

Systematic review of randomised controlled trials (RCTs), or one well-conducted RCT

Harm

Large cohort or pharmacovigilance data

Prognosis

Inception cohort followed from a common early point, such as diagnosis

Diagnosis

Prospective comparison with a blinded reference standard

Aetiology

Converging experimental and observational evidence

Economic or qualitative

A purpose-built study of that kind

The treatment hierarchy

For the effect of a treatment, designs rank from strongest to weakest as follows.

Rank

Design

1

Systematic review of RCTs, with or without meta-analysis

2

Single RCT

3

Cohort study

4

Case-control study

5

Cross-sectional study

6

Case series

  • Randomisation balances measured and unmeasured confounders on average, which puts trials at the top.

  • An RCT ranks above every observational and non-experimental design even when its own result is imprecise or inconclusive, because the ranking rests on randomisation, not on size or significance.

  • A prospective cohort study is the strongest observational design when no trial exists: it measures exposure before the outcome in a defined group with a comparison arm, so incidence and relative risk can be calculated.

Certainty of evidence: the GRADE approach

  • GRADE rates the certainty of a body of evidence from both randomised and observational designs.

  • Each design has a starting level, randomised evidence high and observational evidence low, and either can be moved up or down.

  • Certainty is graded high, moderate, low or very low.

  • Evidence is rated down one level for a serious concern, or two for a very serious one, in risk of bias, inconsistency, indirectness, imprecision or publication bias.

  • Observational evidence can be rated up for a large effect, a dose-response gradient, or residual confounding that would shrink rather than explain the effect.

  • A strong cohort can therefore give more certain evidence than a small, biased trial.

  • Certainty of evidence is separate from strength of recommendation, which also weighs benefit against harm, preferences and practical consequences.

Revision material for the MRCPsych examinations. Not clinical guidance, and not for use in patient care.