13.9.4 Choosing and interpreting common hypothesis tests

Choosing a test

The choice of test follows the type of outcome (categorical, ordinal or continuous), the number of groups, whether observations are paired or independent, and whether continuous data are normally distributed.

• Each test reports its own statistic: ANOVA gives F, the chi-squared test chi-squared, the t-test t, Mann-Whitney U and Kruskal-Wallis H.

Comparing two means

• The paired t-test compares the means of a continuous, normal outcome measured twice in the same people, or in matched pairs such as co-twins. Because the observations are dependent, it analyses the within-pair differences.

• The unpaired (independent samples) t-test compares the means of two unrelated groups by dividing the difference in means by its standard error.

• Change scores: when two independent groups each have a before and after measurement, the mean change is compared between groups with an unpaired t-test, because the pairing applies only within each person.

• The one-sample t-test compares a single sample's mean with a fixed reference value, such as a published population mean.

Comparing three or more means

• One-way ANOVA compares three or more independent groups on a continuous, normal outcome with similar spread in a single F test.

• Running repeated t-tests instead inflates the type I error: four groups give six pairwise tests and a family-wise error near 26%.

• F is the ratio of between-group to within-group variability. With k groups and N observations it has k - 1 and N - k degrees of freedom, so F(2, 87) means three groups and 90 observations.

• A significant F does not say which means differ; post hoc tests then show which pair differs.

• Factorial (two-way) ANOVA tests two or more crossed categorical factors acting on one continuous outcome in independent groups.

• The order of an interaction is the number of factors acting jointly. A significant two-way interaction in a three-factor design means two factors act together, so the effect of one depends on the level of the other.

• Repeated measures ANOVA treats time as a within-subjects factor for one measure taken repeatedly in the same people. Its time by arm interaction tests whether the arms change differently.

Adjusting for baseline

• ANCOVA compares group means on a normal continuous outcome after adjusting for a baseline score or other covariate. This removes a baseline imbalance and usually narrows the confidence interval.

Rank-based tests

• The Mann-Whitney U test is the rank-based counterpart of the unpaired t-test, used for a skewed outcome so that a few extreme values cannot dominate. It also compares an ordinal outcome, such as a coded rating scale, between two independent groups.

• The Wilcoxon signed-rank test ranks the sizes of the within-person differences, keeping their signs. Reading it as a shift in location assumes the differences are roughly symmetrical.

• The Kruskal-Wallis test is the rank-based counterpart of one-way ANOVA.

Categorical outcomes

• The chi-squared test compares observed with expected counts to test association between categorical variables, such as a binary outcome in two independent groups set out as a 2 x 2 table. It applies no distributional assumptions.

• The expected count in each cell is the row total multiplied by the column total, divided by the grand total.

• The degrees of freedom for a contingency table are (rows - 1) x (columns - 1), which gives 1 for a 2 x 2 table.

• The log-rank test comparing k survival curves has k - 1 degrees of freedom, so the number of groups on a Kaplan-Meier plot fixes them.

• A goodness-of-fit chi-squared test compares a binary outcome in one cohort with an expected rate or reference proportion.

Small expected counts

• Cochran's criteria for a valid chi-squared test are that no more than 20% of cells have an expected count below 5 and none below 1; check expected counts, not observed ones.

• Fisher's exact test replaces the chi-squared test for small samples where an expected count falls below 5, calculating exact probabilities without the chi-squared approximation.

Paired binary data

• McNemar's test compares paired or matched binary outcomes, such as detention status in forty patients at admission and at three months. Only the discordant pairs carry the comparison.

• The chi-squared test cannot be used for paired data because it assumes independent observations.

• With fewer than about 25 discordant pairs, use an exact version.

• McNemar's test needs paired nominal counts in a 2 x 2 table, so interval readings such as temperatures can enter only after grouping into categories.

Correlation

Tap to enlarge

Regression and multivariate methods

• Logistic regression models a binary outcome against several predictors at once, through the log odds with binomial errors. It gives an odds ratio for each predictor adjusted for the others.

• Logistic regression does not need normally distributed residuals, whereas t-tests, ANOVA and linear regression do.

• Multivariable models have one outcome and two or more independent variables; multivariate models analyse two or more dependent variables together.

• Path analysis fits an ordered chain of effects, each intermediate variable being an outcome of the one before and a predictor of the next. It splits the total effect into direct and indirect paths.

• Factor analysis reduces many correlated items to a few underlying constructs by their shared variance, whereas cluster analysis groups respondents.

Agreement and reliability

• Cohen's kappa measures agreement beyond chance between two raters sorting cases into unordered categories, whereas a chi-squared test shows only association.

• Kappa is depressed when the prevalence of the rated condition is extreme, even with high raw agreement, because the marginal totals inflate the chance agreement.

• The intraclass correlation coefficient assesses reliability between raters on a continuous score. Its absolute agreement form is lowered both by random scatter and by a consistent gap between raters.