13 Research methods, statistics, critical review and evidence-based practice13.10 Correlation and regression

13.10.14 Polynomial and non-linear regression

2 min read1 figure

Written by Resident Doctor Committee · Published

Try it

Underfitting and overfitting

Interactive

The fitted curve and the dataA degree 2 polynomial fitted to 15 training points, drawn with 80 test points it never saw. Training error 0.39, test error 0.54.Predictor
Training points: the model learns from theseTest points: held back to judge it
Error on the training data
0.39Root mean squared error.
Error on new data
0.54The noise alone gives about 0.35.
Verdict
Too simple: underfittingDegree 4 does best on new data for this sample.
Error against model complexityTraining error falls at every step from degree 1 to 9; error on new data is lowest at degree 4.0.01.02.0123456789Polynomial degree
Training errorError on new data

The data are simulated around a smooth curve with random noise. A straight line misses the curve on old and new data alike. A high-degree polynomial bends to pass near every training point, so its training error keeps falling, but it has learnt the noise and does worse on data it has not seen. Draw a new sample at degree 9 to watch the curve change shape each time, then add training points to see more data tame it.

A polynomial adds powers of the predictor so a straight line can bend. Step the degree up from 2 and compare the error on the training points with the error on new data: past the best degree, extra bends fit the noise rather than the curve.

New in the 2027 syllabus

The College added this topic for exams from January 2027.

Curved relationships

A straight line does not always fit the data. There are two main ways to model a curve.

  • Polynomial regression adds powers of a predictor, such as x squared, so the conditional mean (the average outcome at a given value of x) can follow a curve.

  • Non-linear regression has parameters that enter the equation non-linearly. An example is a rate parameter inside an exponential function, fitted on the original outcome scale.

A polynomial model can still be linear in its coefficients, even though its fitted line is curved in x.

Model

Example

Classification

Straight line

y = a + bx

Linear in coefficients

Quadratic

y = a + bx + cx^2

Linear in coefficients, but curved in x

Exponential

y = a x exp(bx)

Non-linear in b

The degree of a polynomial is its highest power. A quadratic includes x squared, and a cubic includes x cubed.

In a polynomial, the slope changes with x. So the coefficient on the linear term does not, on its own, describe the association across all values of the predictor.

Worked example: a quadratic prediction

Suppose predicted score = 10 minus 4x + x^2, where x is an illustrative numerical predictor.

x

-4x

x squared

1

-4

1

2

-8

4

3

-12

9

  1. At x = 1, predicted score = 10 - 4 + 1 = 7.

  2. At x = 2, predicted score = 10 - 8 + 4 = 6.

  3. At x = 3, predicted score = 10 - 12 + 9 = 7.

The score falls from x = 1 to x = 2, then rises again by x = 3. This is a U-shaped relationship.

The chapter's hypothetical equation produces a U-shaped curve. Its slope changes with x, even though the unknown coefficients of a polynomial model enter linearly. Original illustration drawn in Python for MRCPsychiatrist; see images/ATTRIBUTION.md

Appraising a curved model

  • Overfitting. Higher-order terms can improve the fit to the observed data but may be fitting noise. This gives unstable curves and poor predictions in new data.

  • Extrapolation outside the observed range of the predictor is especially risky, because polynomial curves can turn sharply there.

  • Centring predictors (subtracting a fixed value, such as the mean) can improve numerical behaviour and make the intercept easier to interpret. For a fully specified polynomial, it does not change the fitted predictions.

  • Residual plots and validation help you judge whether the chosen curve describes the data adequately.

  • Transforming the outcome changes the error model and the interpretation. So fitting log(y) is not generally the same as fitting y with a non-linear model.

Revision material for the MRCPsych examinations. Not clinical guidance, and not for use in patient care.

13.10.14 Polynomial and non-linear regression | MRCPsychiatrist