13.3.1 Measures of treatment impact

Event rates

• Control event rate (CER) is the number of events in the control arm divided by everyone randomised to that arm.

• Experimental event rate (EER) is the events in the experimental arm divided by everyone randomised to that arm. Always use the arm's own total, never raw counts or odds.

• Risk of non-response in an arm is 1 minus that arm's response proportion.

• These "rates" are proportions over a fixed follow-up period, not person-time rates.

In this example trial, relapse is counted over twelve months in 100 patients per arm.

• When a trial gives separate 2x2 tables for a beneficial and a harmful outcome, calculate each measure from the table for the outcome it names.

• In a relapse-prevention trial, count sustained remission in the continuation phase over those actually followed through continuation, not over the acute-phase remitters or the number allocated.

Absolute measures

• Absolute measures are differences in percentage points, and ARR is zero whenever the two rates are equal.

• ABI can be read off a Kaplan-Meier curve as the difference between the proportions event-free in the two arms.

Relative risk and relative risk reduction

• Relative risk (RR), or risk ratio, is the risk in the experimental or exposed group divided by the risk in the control or comparison group, each over its own total: 0.28 / 0.40 = 0.70.

• An RR above 1 means increased risk, 1 means no difference, and below 1 means reduced risk.

• RR is a ratio of non-negative risks, so it cannot be negative; attributable risk is a difference and can be zero or negative.

• RR needs the risk in each group, so cohort studies, randomised trials and cross-sectional studies can give it, but case-control studies cannot.

• Relative risk reduction (RRR) is ARR divided by CER, which equals 1 minus RR: 0.12 / 0.40 = 0.30. It is the share of the control group's risk that treatment removes.

• Relative benefit increase (RBI) is the same idea for a good outcome, ABI divided by CER: 0.12 / 0.60 = 0.20.

• RRR, or RBI for a good outcome, is the summary of a dichotomous treatment effect that efficacy randomised trials report most often.

The same 12-point difference gives an RRR of 30% for relapse but an RBI of 20% for remaining well, because the denominators differ, so always name the event.

Odds ratio

• Odds are events divided by non-events, not by the group total: experimental odds are 28 / 72 = 0.39.

• Odds ratio (OR) compares the odds in one group with the odds in another: (a x d) / (b x c) = (28 x 60) / (72 x 40) = 0.58.

• An OR below 1 is protective and above 1 means increased risk.

• The OR approximates the RR when the outcome is rare and moves further from 1 than the RR as the outcome becomes common (0.58 against 0.70 here). An event rate of 10% is only a rough guide to "rare".

• Logistic regression estimates ORs, including in cohort and trial data.

• In a table of risk factors, the size of the OR shows the strength of association: the largest OR whose interval excludes 1 marks the strongest predictor.

Tap to enlarge

Values that mean no difference

Number needed to treat and number needed to harm

• Number needed to treat (NNT) is 1 divided by ARR, or by ABI. Calculate it from the absolute difference in event rates, never from a relative measure or one arm's rate.

• Round NNT up to the next whole patient, not to the nearest whole number: 1 / 0.12 = 8.33 gives 9.

• Number needed to harm (NNH) is 1 divided by ARI for an adverse outcome, the number treated for one extra patient to be harmed: 1 / 0.15 = 6.67, rounded up to 7.

• To back-solve the treated arm, add 1 / NNH to the CER to get its event rate.

• NNT for an individual patient is 1 / (baseline risk x RRR), so it falls as baseline risk and RRR rise.

• NNT is the best measure for explaining benefit to a patient, as the number of people treated for one to benefit; a smaller NNT is more favourable.

• An NNT belongs to its time horizon: NNTs over one and five years are not directly comparable, and an NNT of 9 over twelve months cannot be weighed against an NNH of 7 over four weeks.

Tap to enlarge

Confidence intervals

• A confidence interval is a range around the estimate at a stated confidence level. It shows precision and is narrow when the sample is large or variation small.

• ARR, RRR, NNT or NNH, and the confidence interval each answer a different question about the same data.

• The 95% confidence interval of the NNT is the reciprocal of each limit of the ARR interval, with the limits swapping places: ARR 0.12 (0.03 to 0.21) gives NNT 9 (5 to 34).

• When the ARR interval crosses zero, say minus 0.05 to 0.29, the NNT interval runs from NNT 4 through infinity to NNH 20. It includes no effect and does not establish equivalence.

Tap to enlarge

Absolute against relative measures

• Absolute measures (ARR, NNT) change with the baseline event rate, whereas ratio measures (RR, RRR, OR, hazard ratio) are more stable across populations.

• Relative measures are usually much larger than the absolute difference, especially when events are uncommon, so they can make a small benefit look large.

With RR held at 0.70:

Attributable and population measures

• RR is the ratio of the same two incidences that attributable risk subtracts.

• Among modifiable risk factors for dementia, hearing loss has the largest weighted population attributable fraction, and the combined weighted fraction is about 40%.

Continuous outcomes

• The ANCOVA mean difference differs from both the raw difference in follow-up means and the difference in change scores.

• Conventional benchmarks for Cohen's d or Hedges' g are about 0.2 small, 0.5 moderate and 0.8 large.

Time-to-event outcomes

• The hazard ratio (HR) is the effect size of Cox proportional hazards analysis, which uses event timing and censored follow-up. OR and RR suit binary outcomes at one time point.

• An HR of 0.63 in a self-controlled analysis means a 37% lower hazard in treated periods than in the same people's untreated periods.

• Results stratified by individual drug within a class can differ in direction across outcomes, so read them agent by agent and outcome by outcome. In one such analysis clozapine had an HR below 1 for all three causes of death.

Cost-effectiveness

• The incremental cost-effectiveness ratio (ICER) is the difference in cost divided by the difference in effect. Against no intervention, it is the total programme cost divided by the events prevented.