13 Research methods, statistics, critical review and evidence-based practice13.2 Epidemiological studies
13.2.3.2 Appraising prognosis studies
3 min read2 figures
Written by Resident Doctor Committee · Published
Framing the question
Question | Estimates | Validity turns on |
|---|---|---|
Overall prognosis | Average risk over a stated period | How the cohort was assembled |
Prognostic factor | How one variable shifts that risk | Confounding and measurement |
Prognostic model | Individual risk from several variables | Performance outside the development data |
A prognostic question names the population, starting point, outcome and horizon; "20% readmission" means nothing without a time period.
A prognostic association does not show causation, or that changing the predictor improves outcomes.
The inception cohort
An inception cohort starts at a clearly defined early stage, avoiding selection by illness duration; an inconsistent time zero causes immortal time and survivor bias.
Tertiary clinic and long-stay samples miss people who recovered, died early or disengaged.
AESOP-10 followed 557 people with first-episode psychosis in English catchments for ten years and found a more varied course than uniform deterioration.
Measuring factors and outcomes
Outcome assessors should be blind to predictor status, and retrospective predictor coders blind to outcomes.
Unequal surveillance, monthly in one group and annually in another, changes detection rates without changing true event rates.
Follow-up and censoring
Survival methods assume independent censoring, so loss linked to worsening or recovery biases estimates in either direction.
If 15% are known to recover and 30% are of unknown status, recovery lies between 15% and 45%, a bound rather than a confidence interval.
Fixed loss thresholds, such as 5% or 20%, are only prompts, because bias depends on why outcomes are missing.
How prognostic factor models mislead
Adjusting for post-baseline mediators changes a causal interpretation, though time-updated prediction may legitimately use later information.
Overfitting follows data-selected cut-points and heavy predictor screening; a "critical" duration of six months in one study and twelve weeks in another may reflect this.
Sample adequacy depends on events, not total participants: 800 people with 30 relapses cannot support twenty candidate predictors.
Reading the numbers
A hazard is the instantaneous event rate among those still at risk.
A Cox proportional hazards model assumes a constant hazard ratio over time, which inspecting curves cannot fully confirm.
The same hazard ratio means different things at baseline risks of 1% and 40%, so absolute risk at a stated horizon should be reported.
A hazard ratio is not a risk ratio
Under proportional hazards, S1(t) = S0(t) ^ HR, where HR is the hazard ratio. With HR 1.8 and a reference twelve-month readmission risk of 20%, S1 = 0.80 ^ 1.8 = 0.669.
Comparison risk is about 33%, giving a risk ratio of 1.66 and an absolute difference of 13 percentage points, not "80% more readmitted".
Model performance
Discrimination is how well a model ranks risk; calibration is how closely predicted probabilities match observed ones.
A c statistic of 0.75 does not establish clinical usefulness, and a poorly calibrated model is not automatically safe even for prioritising review.
Validation
Approach | Method | Estimates |
|---|---|---|
Bootstrapping | Repeats the whole development procedure in resamples | Optimism, and informs shrinkage |
Cross-validation | Fits and tests across nested folds | Performance on held-out data |
External validation | Applies the fixed model to new patients, periods or settings | Transportability, discrimination and calibration |
Development performance is optimistic, and performance commonly but not always falls on external validation.
TRIPOD+AI (2024) guides reporting of prediction models and PROBAST+AI (2025) assesses bias and applicability; complete reporting does not mean low bias.
Suicide and violence prediction
A 2019 review of 64 models in 17 studies, with over 14 million participants, found high classification accuracy but a suicide positive predictive value generally of 0.01 or less.
Accuracy is not discrimination and can be high simply because the outcome is rare.
Low prevalence produces many false positives at ordinary specificity; whether most suicides fall in the low-risk group depends on sensitivity.
NICE NG225 advises against scales or global risk categories to predict suicide or repeated self-harm, or to decide treatment or discharge.
Violence tools need context-specific interpretation, not automatic pathways triggered by a score.
Applicability
Treatment era, illness stage, referral pathway and baseline risk decide whether a cohort fits current services, and older cohorts may need recalibration.
Readmission partly reflects service practice, while employment, relationships and function may matter as much to the patient.