Study guides · Intermediate

Samples, Estimation, Hypothesis Tests, and Evaluation

Connect sampling, standard errors, confidence intervals and tests without confusing uncertainty with a guarantee.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. Separate sampling bias from sampling variation
  2. Distinguish spread from uncertainty of an estimate
  3. State what a test evaluates
  4. Make the comparison reproducible
  5. Ask Logeia about this articleCheck your understanding

Separate sampling bias from sampling variation

A sample mean summarizes the observations collected, not an automatically known population mean. Repeated random samples make it possible to reason about sampling variation. Selection bias is different: collecting only daytime users may omit a distinct nighttime workload, however many daytime observations are added. For a response-time study, first define the population of requests and the sampling procedure. A correctly computed average can still answer the wrong question. Independence, representative sampling and a stable target population are assumptions to examine, not properties created by applying a statistical formula.

Distinguish spread from uncertainty of an estimate

The sample standard deviation s describes the spread of individual observations. Under appropriate independent, identically distributed sampling, s divided by the square root of n estimates the standard error of the sample mean. With n=100 and s=20 milliseconds, that standard error is two milliseconds. A confidence interval uses an appropriate quantile, often from a t distribution; a large-sample normal approximation uses approximately 1.96 for a two-sided 95 percent interval. Autocorrelated measurements do not supply the same information as independent ones, so request count alone cannot justify the usual calculation.

State what a test evaluates

A p-value is computed under a null hypothesis and a specified testing model. It is not the probability that the null hypothesis is true. A statistically significant one-millisecond API improvement might be operationally unimportant, while a nonsignificant difference does not establish equivalence. Report an effect estimate, uncertainty interval and measurement conditions alongside the test. Repeatedly checking results and stopping at a convenient moment, or selecting the smallest p-value from many metrics, changes the interpretation. A decision rule needs to match the actual data-collection and analysis procedure rather than an idealized one used only in the report.

Make the comparison reproducible

When comparing two implementations, record load, warm-up, execution order and error rates. Means can hide a small set of very slow requests, making quantiles useful, but a p99 estimate needs enough tail observations to be stable. Specify an outlier rule before inspecting which implementation benefits from removal. Failure-related delays are often part of the behavior being measured, not disposable noise. Finally, explain which hardware, time period and user population the estimate can reasonably represent. Reproducibility requires both a calculation and the operational context that generated its inputs.

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Prepare for AI systems · STEP 11

Choose a learning path