13 Research methods, statistics, critical review and evidence-based practice13.5 Diagnostic studies

13.5.9 Reliability

Creator: Resident Doctor Committee
Published: 2026-10-14T21:05:44.632Z

3 min read1 figure

Written by Resident Doctor Committee · Published 14 October 2026

Try it

Kappa calculator

Interactive

Two raters assessing the same people. Change any count and watch what chance agreement does to kappa.
	Rater 2: case	Rater 2: not a case	Rater 1 total
Rater 1: case	Both raters say case	Rater 1 case, rater 2 not a case	45
Rater 1: not a case	Rater 1 not a case, rater 2 case	Both raters say not a case	30
Rater 2 total	44	31	75

Observed agreement90.7%

Expected by chance alone51.7%

Kappa: the share of the agreement chance cannot explain0.81

Kappa: 0.81Almost perfect
Observed agreement: 90.7%The proportion of assessments the raters called the same way.
Expected agreement: 51.7%What their own marginal habits would produce if they guessed independently.
Disagreements: 9.3%7 of 75 assessments.

Raise the count of agreed non-cases and observed agreement climbs while kappa falls: when almost nobody has the condition, two raters agree easily by chance, so there is little left for kappa to credit them with. Kappa is undefined when chance agreement reaches 100 %.

Observed agreement 0.91, expected agreement 0.52, kappa 0.81. Type in the delirium table below (20, 8, 7 and 65) and check that kappa falls to 0.62.

Reliability, precision and validity

Reliability is the consistency of a measure when the thing measured has not changed, and it depends on precision, the closeness of repeated measurements to each other.
Accuracy, the counterpart of validity, is the closeness of a measurement to the true value.
A tight cluster of results well away from the reference value shows precision with a systematic offset, like a scale that always reads 3 kg too high: reliable but inaccurate.

A two by two grid of dartboards, with precise and imprecise columns and accurate and inaccurate rows. Under each board, a curve shows how the shots spread around the reference value. — Look at the bottom left board: a tight cluster away from the reference value is precise but inaccurate, the reliable but inaccurate pattern that systematic error produces. Arbeck, CC BY 4.0, via Wikimedia Commons

Error	What it does to results	What it threatens
Random error (imprecision)	Scatters them without direction around the true value	Reliability
Systematic error, confounding and bias	Pushes them one way	Validity

Reliability and validity are separate families of psychometric properties, and test-retest, inter-rater, split-half, parallel-forms and internal consistency all belong to reliability only.
Reliability limits validity but does not confer it.

Types of reliability

Type	Design	What it shows
Test-retest	Same instrument given to the same people on two occasions while the trait is unchanged	Stability over time
Parallel-forms (alternate-forms)	Two equivalent versions of one instrument completed by the same people	Agreement between versions, allowing repeat testing without a practice effect
Internal consistency	The items of one scale, given once	Whether the items measure the same construct
Split-half	One test, given once, divided into two halves whose scores are correlated	Agreement between the halves
Inter-rater	Different raters score the same material with the same instrument	Agreement between raters
Intra-rater	One rater scores the same fixed material on two occasions	Agreement of a rater with themselves

Cronbach's alpha is the standard index of internal consistency for a multi-item scale.
A scale meant to measure one construct should be homogeneous, so each item shows a positive item-total correlation with the rest of the scale.
A very short retest interval inflates agreement through memory, and a long one may include real change.
In qualitative analysis, more than one researcher codes the data independently, because the threat to reliability lies in one person's interpretation.

Statistics for agreement

Statistic	When it is used
Cohen's kappa	Two raters making categorical (nominal) judgements; every disagreement counts as equally wrong
Weighted kappa	Two raters making ordinal ratings; near-agreement gets partial credit
Intraclass correlation coefficient	Continuous ratings
Chi-squared and correlation	Association, not agreement

Scores that are consistently 5 points apart correlate perfectly yet never agree.

Calculating Cohen's kappa

Observed agreement is the raw proportion of all cases rated the same way, summing both concordant cells over the total, and it is a different figure from kappa.
Expected agreement is the agreement chance would produce, worked out from each rater's marginal rates.
Kappa is observed agreement beyond chance divided by the maximum agreement available beyond chance: kappa = (observed - expected) / (1 - expected).

Two raters assess 100 patients for delirium:

Rater A rating	Rater B: present	Rater B: absent	Rater A total
Present	20	8	28
Absent	7	65	72
Rater B total	27	73	100

Observed agreement = (20 + 65) / 100 = 0.85.
Expected agreement = (0.28 x 0.27) + (0.72 x 0.73) = 0.6012.
Kappa = (0.85 - 0.6012) / (1 - 0.6012) = 0.62.

Interpreting kappa

Kappa runs from -1 to +1: +1 is complete agreement, 0 is agreement equal to chance, and negative values are worse than chance.
A high kappa means strong inter-rater agreement beyond what chance alone would produce.

Landis and Koch grade kappa as follows:

Kappa	Agreement
0 to 0.20	Slight
0.21 to 0.40	Fair
0.41 to 0.60	Moderate
0.61 to 0.80	Substantial
0.81 to 1.00	Almost perfect

Revision material for the MRCPsych examinations. Not clinical guidance, and not for use in patient care.