13 Research methods, statistics, critical review and evidence-based practice13.5 Diagnostic studies
13.5.10 Appraising diagnostic accuracy studies
2 min read1 figure
Written by Resident Doctor Committee · Published
Framing the question
Accuracy figures mean little without the population, setting, index test, threshold, target condition and reference standard.
Sampling and spectrum
Design | Recruitment | Effect on accuracy |
|---|---|---|
Single-gate | One clinically defined population, ideally enrolled consecutively | Can be representative |
Two-gate | Cases and non-cases recruited through separate routes | Can exaggerate accuracy; predictive values reflect an artificial prevalence |
Spectrum matters: telling severe dementia from healthy volunteers is easier than classifying mild or ambiguous memory clinic presentations.
Reporting versus risk of bias
STARD (2015, 30 items) is the reporting guideline specific to diagnostic accuracy studies.
QUADAS-2 is a risk-of-bias tool covering patient selection, index test, reference standard, and flow and timing; QUADAS-3 updates it.
Complete reporting can reveal a biased study but cannot make it unbiased.
Biases in conduct
Bias | What goes wrong | Safeguard |
|---|---|---|
Review bias | The index test is read knowing the reference result (test review) or the reverse (diagnostic review) | Apply index test and reference standard independently, each interpreted blind to the other |
Partial verification | Only some participants, often selected by index result, get the reference standard; sensitivity is inflated and specificity and predictive values distorted | Apply the reference standard to every participant regardless of the index result, ideally by an assessor blind to it |
Differential verification | Participants receive different reference methods | Report who received which standard |
Incorporation bias | The index test contributes to the reference diagnosis, so agreement is partly circular | Exclude the index result from the reference decision |
Sensitivity, specificity and ROC curves
Sensitivity in a memory clinic example is 113 of the 120 people with dementia screening positive (0.94); specificity is 249 of the 280 without dementia screening negative (0.89).
Predictive values change with prevalence and spectrum.
A ROC curve plots sensitivity against 1 minus specificity across all thresholds.
1 minus specificity is the false positive rate, the people without the condition whom the test calls positive, and forms the horizontal axis.
The diagonal line, where sensitivity equals 1 minus specificity, represents chance-level discrimination, with an area under the curve of 0.5.
The optimal cut-off lies at the elbow or shoulder of the curve: nearest the upper-left corner, furthest above the diagonal (largest Youden index), where sensitivity and specificity are jointly best.
Thresholds should be prespecified; a threshold chosen from the study data looks optimistically good and needs independent validation.