13 Research methods, statistics, critical review and evidence-based practice13.2 Epidemiological studies
13.2.1.1 Bias and error
8 min read6 figures
Written by Resident Doctor Committee · Published
Sources of bias and strategies to reduce them
Bias and confounding
Selection bias is a systematic difference between the people studied and the population the results describe, arising from how participants enter or stay in a study.
Information (measurement) bias is error in measuring the exposure or outcome in people already enrolled.
Confounding arises when a third variable linked to both the exposure and the outcome is left unstratified and unadjusted.
Confounding differs from selection and ascertainment biases, which concern who is recruited or how cases are found.
Selection bias
A larger sample does not remove the selection mechanism.
Form | How it arises | Remedy |
|---|---|---|
Self-selection | Recruiting by advertisement, an open invitation to a voluntary survey, or a convenience sample at a non-representative venue | Suitable sampling |
Referral | Specialist services over-represent severe cases | Define the source population in advance |
Inappropriate controls | An eligibility criterion applies to controls but not cases, or controls come from an unrepresentative service | Draw controls from the population that produced the cases |
Neyman (prevalence-incidence) bias | Cases are long-standing survivors, because those who recovered or died early were gone before sampling | Recruit incident cases |
Inappropriate controls undermine internal validity (whether the association is true within the sample), not external validity (generalisability), and can pull the odds ratio towards 1.
A general surgical sample may be unsuitable as controls for clozapine-related agranulocytosis.
In Neyman bias, an exposure linked to long illness is over-represented among cases.
Berkson's bias (admission rate bias) is a selection bias of hospital-based samples: people with two conditions or with the exposure are admitted more readily.
These unequal admission probabilities create or distort an association among inpatients that is absent in the community.
A self-selected sample is ordinary selection bias, not Berkson's bias, which is the hospital-admission form.
Consecutive recruitment cuts down picking and choosing within a service but leaves that service's referral biases untouched.
Loss to follow-up and non-response
Attrition bias is systematic loss of participants over follow-up, which can hide excess mortality or manufacture a difference between arms.
Attrition occurs when untraced people have died or deteriorated, or when people drop out of a demanding intervention.
Attrition is limited by several contact routes, tracing, comparing leavers with stayers and checking retention in each arm.
The size of a loss, even 30%, does not settle the direction of bias; the reasons, the timing and sensitivity analyses do.
A completer analysis overstates benefit if non-responders withdraw and understates it if improving patients leave.
Intention-to-treat analysis keeps the original allocation but does not by itself recover missing outcomes.
Collider bias arises from conditioning on a common consequence of the exposure and of other outcome-related factors.
Non-response bias arises because people who do not reply differ systematically from responders, so results describe responders and usually lower a prevalence estimate.
Non-response is handled by following up non-returners and comparing the known characteristics of responders and non-responders.
Information bias
Non-differential misclassification measures both groups equally badly and often waters down a simple association, though not always.
Differential misclassification differs between groups and can move an effect in either direction.
Measurement error that dilutes a real harm makes a null finding less reassuring, not more.
Bias | Mechanism | Remedy |
|---|---|---|
Recall bias | Cases and controls differ systematically in how completely they remember and report a past exposure | Records made at the time |
Observer (detection, measurement) bias | An outcome assessor who knows the allocation measures or classifies the outcome differently | Blind the outcome assessors |
Diagnostic suspicion bias | A rater who knows each person's exposure probes harder or scores more readily in the exposed, inflating an association | Blinded, standardised assessment |
Surveillance (detection) bias | Outcome ascertainment is more intense in one exposure group, so more outcome is found there without a true difference | Comparable ascertainment |
Recall bias is the bias most characteristic of the retrospective case-control design: differential maternal recall can exaggerate a link between obstetric complications and psychosis.
Scheduled renal testing during lithium treatment finds genuine disease but also shifts ascertainment towards the treated group.
Verification (work-up) bias arises in diagnostic accuracy studies when the index test result decides who receives the reference standard.
Verification bias cannot arise in an aetiological case-control study, whose threats are selection and information biases.
Bias in randomised trials
Safeguard | What it protects against |
|---|---|
Randomisation | Confounding |
Allocation concealment | Selection bias |
Double blinding of participants and outcome raters | Expectation (observer-expectancy) bias |
Blinding outcome assessors | Observer bias |
Blinding treating clinicians | Cointervention (performance) bias |
Cointervention (performance) bias arises when staff who know the allocation give one arm extra care, such as attention or advice.
Randomisation and allocation concealment cannot prevent cointervention bias because they act only at entry.
Contamination is when control participants are exposed to or adopt the experimental intervention, usually through shared staff or settings, biasing the effect towards the null.
Cluster randomisation is designed to prevent contamination, but it needs a design and analysis to suit.
A clustering (trial-centre) effect occurs when treatment is allocated or delivered by centre, because patients within a centre have correlated outcomes.
Analysing clustered patients as independent overstates precision and inflates the type 1 error; randomising individual patients rather than centres avoids it.
Control-arm patients often do better than similar patients outside trials because of selective eligibility, placebo expectancy, protocol-driven monitoring and specialist centres, but not because of randomisation.
A fall in the usual-care arm over time signals non-specific effects and an unmatched comparator, especially when the intervention arm receives extra contact that usual care does not match.
Expectation and observation effects
The Hawthorne effect is a change in participants' behaviour or performance because they know they are being studied, independent of the intervention.
The Hawthorne effect touches volitional behaviour, not laboratory, electrocardiogram or mortality outcomes.
The Pygmalion effect (experimenter expectancy or Rosenthal effect) is a change in subjects' responses towards what the researcher or practitioner expects, transmitted through their differential behaviour.
The Hawthorne effect comes from being observed; the Pygmalion effect comes from the observer's expectations.
Change over time
Regression to the mean is the drift of a repeat measurement back towards the average in people selected for an extreme score.
Only a randomly allocated control arm, which regresses equally, removes regression to the mean from the estimated effect.
In an uncontrolled before-after study, a fresh pre-intervention baseline exposes regression to the mean.
The practice (testing) effect is a rise in scores from sitting the same or a similar test more than once, independent of any intervention.
Screening biases
Bias | What happens |
|---|---|
Lead-time bias | Screening moves diagnosis earlier, so survival measured from diagnosis lengthens even when the time of death is unchanged |
Length-time bias | A single screening round catches slow-growing tumours more often because they spend longer in the detectable phase, so screen-detected cases carry a better prognosis whatever screening achieves |
Publication and reporting bias
Publication bias arises when whether a study reaches print depends on its findings, so positive trials are over-represented.
A review built from journal papers alone overstates benefit; searching trial registries and regulatory files reduces publication bias.
An asymmetric funnel plot with an empty region where small null trials should sit suggests publication bias.
Selective reporting happens when researchers replace the primary outcome after seeing results, report only favourable subgroups or time points, or try models until one is significant.
Selective reporting needs no deliberate deception to exaggerate the evidence; it is detected by comparing the report with its prespecified protocol and analysis plan.
Analyses should be prespecified and registered, with deviations reported.
CONSORT and STROBE make reporting more transparent but do not prove the underlying design was sound.
Errors of inference
The ecological fallacy is drawing a conclusion about individuals from area-level data.
A type 1 error is a false positive, such as a chance significant finding among many comparisons each judged at the 5% level.
A type 2 error is failing to reject a false null hypothesis, as when too small a sample lacks the power to detect a real effect.
Systematic error, random error and validity
Two kinds of error
Feature | Systematic error (bias) | Random error (chance) |
|---|---|---|
Origin | Problems in design, conduct, measurement, analysis or reporting | Chance variation between samples |
Pattern | Consistent, reproducible shift in one direction | Unpredictable scatter around the true value |
Repetition or a larger sample | Recurs on every measurement and persists however many people are recruited | Shrinks, so estimates become more precise and reliable |
What it limits | Validity | Precision |
Precision concerns random uncertainty, while validity concerns whether a result is credible and applicable.
Bias falls into selection bias and information (measurement) bias, and confounding distorts estimates by a separate mechanism.
More confident and more wrong
The standard error falls with the inverse square root of sample size, so four times as many participants roughly halves it.
Pooling more biased observations narrows the interval around the wrong value: help-seeking recruitment estimates depression prevalence at 24% (true value about 10%) whether 200, 2,000 or 20,000 people take part.
A conventional confidence interval reflects random uncertainty and does not account for selection or measurement bias.
Internal and external validity
Type | Question it asks |
|---|---|
Internal validity | Is the estimate credible, given selection, measurement, confounding and other threats? |
External validity | Does the result apply to another population, setting or delivery context? |
Measurement validity | Does the instrument measure the intended construct? |
An antidepressant trial that excludes physical illness, suicidality and substance misuse may not apply to an older patient with those problems, which is a limit of external validity.
Assess internal validity before applying a result elsewhere.