13 Research methods, statistics, critical review and evidence-based practice13.3 Experimental studies
13.3.1.13 Appraising therapy, harm and aetiology studies
4 min read2 figures
Written by Resident Doctor Committee · Published
Validity and trial design
Internal validity is the property that lets a study attribute an observed effect to the intervention or exposure rather than to confounding, bias or chance.
Explanatory trial | Pragmatic trial | |
|---|---|---|
Aim | Efficacy under tight experimental control | Effect under the conditions of everyday care |
Favours | Internal validity | External validity |
Eligibility, delivery and comparator | Narrow and controlled | Mirror everyday care |
Explanatory trials depart from routine practice on several dimensions at once: selection, allocation, blinding, placebo, follow-up and endpoint.
Blinding of patient and prescriber is the departure that has no counterpart in routine care.
Observational designs
Design | Strength | Main threat |
|---|---|---|
Randomised trial | Allocation is independent of prognosis, so measured and unmeasured confounders balance | Too small and short for uncommon or delayed harm |
Cohort | Exposure is recorded first, which secures temporality | Confounding by indication and differential loss to follow-up |
Case-control | Efficient for rare or long-delayed outcomes | Choice of controls and how exposure is ascertained |
Confounding by indication is the principal threat to an observational register cohort comparing two treatments, because without randomisation nobody controls who receives which drug.
Before reading an association as causal, such as benzodiazepines and later dementia, consider reverse causation (prodromal insomnia prompts prescribing), confounding, selection and detection bias.
Reporting a randomised trial
The CONSORT statement recommends a participant flow diagram in the report of a randomised trial.
The follow-up stage of the flow diagram shows attrition: the losses to follow-up and discontinuations in each arm, with reasons.
Endpoints
A patient-important outcome counts as benefit in itself.
A surrogate endpoint is measured in place of the patient-centred outcome the study aims to change, and matters only because it is expected to predict that outcome.
A secondary outcome answers a further, separate hypothesis, which is what distinguishes it from a surrogate.
A composite endpoint needs fewer participants and shorter follow-up, but it hides the fact that patients value its components differently.
Judging the result
Clinical significance is judged by the between-arm difference against a threshold set in advance, not by the p-value or by change within one arm.
In one severity meta-analysis, the drug-placebo gap reached an effect size d of 0.50 only above a baseline Hamilton Rating Scale for Depression score of about 28.
The number needed to treat is 1 divided by the absolute risk reduction and needs a time horizon: control risk falling from 32% to 24% gives 1 / 0.08, rounded up to 13 over twelve months.
A non-significant intention-to-treat comparison of the primary outcome means no demonstrable difference in effect between arms; a difference in drop-out between the same arms is a separate finding.
Subgroups, repeated measures and small studies
A subgroup finding is most believable when the subgroup hypothesis was stated in the protocol before any outcome data existed.
Many subgroups, a failed primary result and failed replication each weaken a subgroup finding.
In a repeated-measures trial, significance is read at each assessment point, so an effect can be present during treatment and lost after withdrawal.
Covariate-adjusted comparisons are not confounded by baseline severity, and group means say nothing about individual patients.
A small, unpowered sub-study of a comorbid group supports the feasibility and acceptability of the therapy for the condition under test, not proven efficacy.
A small pilot trial of a psychological treatment can collect both qualitative and quantitative data, but lacks blinding, representativeness, durable follow-up and statistical power.
Stopping early and attrition
The reason most often reported for stopping a randomised trial early is an apparently large benefit at interim analysis.
Such trials tend to overstate the true effect: few events have accrued at an interim look, and stopping on a favourable chance fluctuation selects random highs that would have regressed.
A positive result is robust to attrition only if it survives worst-case sensitivity analysis, which imputes the outcome least favourable to the intervention for every missing participant.
Why a trial cannot show a drug is safe
The rule of three says that zero events among n participants gives an approximate one-sided 95% upper bound on risk of 3/n, so zero agranulocytosis cases in 300 gives about 1% over the observed period.
Yellow Card spontaneous reports to the Medicines and Healthcare products Regulatory Agency detect signals but cannot estimate incidence, because reporting is incomplete and there is no exposure denominator.