Concepts and theorems · Intermediate

Gradient noise

Calculate the mean and variance of sampled gradients and distinguish batch-size effects from biased sampling and reproducibility.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. Identify the target gradient
  2. Compute a two-example distribution
  3. Explain the batch-size calculation
  4. Distinguish variance from bias
  5. Connect gradient noise to updates
  6. Use reproducibility without hiding variability
  7. Ask Logeia about this articleCheck your understanding

Identify the target gradient

Consider mean training loss L(w), the average of differentiable per-example losses. Its full-data gradient is the average of their gradients. A uniformly sampled example supplies an estimate that varies with the selected example. This sampling variation is distinct from measurement error in inputs or floating-point rounding. Hold w fixed while analyzing it. If parameters move between measurements, the observed difference combines sample selection with evaluating the objective at different locations. Naming the target gradient and the source of randomness prevents several unrelated forms of variation from being grouped under one vague label.

Compute a two-example distribution

At a fixed parameter, suppose two per-example gradients are one and three. Uniformly selecting one gives mean two and variance one. The full-data average gradient is also two, so this estimator is unbiased even though individual draws never equal two. The values are invented to isolate the mechanism rather than report a measured training run. If examples with opposite-signed gradients are used instead, one sampled update can point differently from the full-data gradient. Unbiasedness is an expectation over the sampling procedure, not a guarantee about each realized update or loss change.

Explain the batch-size calculation

Averaging B independent draws with replacement reduces finite variance by a factor of B. In the previous example, a batch of four has gradient-estimator variance one quarter. This statement uses an average, not a sum. Switching to summed batch loss changes gradient scale. Sampling without replacement introduces a finite-population adjustment, and correlated observations require their covariance to be considered. Repeated measurements from the same user are not automatically independent examples. State sampling and reduction conventions before attributing a change in gradient behavior solely to batch size.

Distinguish variance from bias

If the gradient-three example is selected with probability 0.9 and the gradient-one example with probability 0.1, the expected estimate is 2.8 rather than two. Larger batches reduce spread around 2.8; they do not automatically recover the uniformly weighted objective. When intentionally changing sampling weights, state whether the target objective also changes or whether a correction is applied. The same issue arises when emphasizing difficult examples. Reducing estimator variance and estimating the intended gradient are separate requirements. An extremely reproducible low-variance computation can still optimize the wrong weighted objective.

Connect gradient noise to updates

For update w next=w−eta times the estimated gradient, conditional update variance at fixed w is eta squared times gradient-estimator variance. Increasing batch size and increasing learning rate can therefore push update variability in opposite directions. This does not imply that multiplying learning rate by batch size always preserves training: curvature, update count, and data reuse change as well. Record learning rate, batch size, updates per pass through the data, and whether loss is summed or averaged. Those details determine whether two trajectories are meaningfully comparable.

Use reproducibility without hiding variability

A fixed seed helps reproduce the same sample sequence but does not show that sampling noise is absent. Parallel execution and implementation differences can also prevent identical outputs despite equal seeds. For practice, repeatedly sample gradients one and three with replacement and compare batch means for B=1 and B=4. Their variances should be near one and one quarter over many repetitions. Then bias the sampling probabilities and report the mean as well. Completion requires explaining reduced variability separately from a shifted expectation, rather than treating a stable-looking learning curve as proof of an unbiased gradient.

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Prepare for AI systems · STEP 14

Choose a learning path