Concepts and theorems · Intermediate

Bias and variance

Compare predictions across repeated training samples and decompose squared prediction error into bias, variance, and observation noise.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. Vary the training sample
  2. State the squared-loss assumptions
  3. Calculate a small example
  4. Avoid a universal complexity story
  5. Recognize what real data cannot reveal
  6. Connect the concept to an experiment
  7. Ask Logeia about this articleCheck your understanding

Vary the training sample

Bias and variance concern predictions across repeated training samples, not only one fitted model. Hold an input x fixed, denote its true mean response by f(x), and imagine fitting the procedure on different samples. Bias compares the average prediction with f(x); variance measures fluctuation around that average prediction. One observed prediction error does not separately reveal both quantities. State what stays fixed and what is resampled before using the terminology. Otherwise training residuals, test errors, and variability between fitted models can be incorrectly treated as interchangeable measurements.

State the squared-loss assumptions

Assume a fresh response Y=f(x)+epsilon with conditional mean-zero noise and conditional noise variance sigma squared. Let that fresh noise be independent of the training sample. Expected squared prediction error decomposes into squared bias, prediction variance, and noise variance. The expectation averages over fitted models and fresh observations under these assumptions. It is a squared-loss identity, not an automatic formula for classification accuracy or every other loss. Improving measurement quality, changing a model class, and collecting more training data can affect different parts of the decomposition and should not be described as the same intervention.

Calculate a small example

At a fixed x, let the true mean be two and let predictions be one, two, or three with equal probability across training outcomes. Average prediction is two, bias is zero, and prediction variance is two thirds. If independent fresh-observation noise has variance one half, expected squared error is seven sixths. Dividing by three here averages over an explicitly specified three-state distribution; it is not the denominator used for an unbiased sample-variance estimator. Distinguish computing an expectation in a known toy distribution from estimating an unknown variance using a finite set of actual training runs.

Avoid a universal complexity story

Flexible models can fit training data closely while responding strongly to changes in the sample. Restrictions or regularization can reduce prediction variability while increasing bias. However, increasing model size does not universally increase variance or decrease bias in every practical training setting. Data volume, optimization, features, and implicit constraints may change too. Use the decomposition as a way to formulate questions, not a guaranteed narrative about every learning curve. Reduced training error alone cannot establish improved performance on unseen observations, especially when the evaluation distribution differs from the training distribution.

Recognize what real data cannot reveal

In real data, f(x) and noise variance are usually unknown, so exact bias and variance are not directly observable. Variation across random seeds or resamples depends on which factors change. Altering initialization alone is not the same experiment as drawing fresh training samples. Fix the evaluation inputs and record whether training samples, initialization, or preprocessing vary. Bootstrap estimates introduce their own empirical-distribution and independence assumptions. Report the variability that was actually measured rather than assigning every change in test performance to a separately identified bias or variance component.

Connect the concept to an experiment

Compare a procedure that always predicts 1.5 when the true mean is two. Its variance is zero, squared bias is 0.25, and with noise variance 0.5 its expected error is 0.75. Bias increased relative to the previous toy procedure, yet total error decreased. This is why bias should not be treated as an isolated quantity to eliminate at all costs. In practical regularization comparisons, keep training, validation, and final evaluation distinct and compare held-out loss. Then connect the decomposition to the separate mechanisms of regularization and noisy gradient estimation.

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Prepare for AI systems · STEP 15

Choose a learning path