Separate sampling bias from sampling variation
A sample mean summarizes the observations collected, not an automatically known population mean. Repeated random samples make it possible to reason about sampling variation. Selection bias is different: collecting only daytime users may omit a distinct nighttime workload, however many daytime observations are added. For a response-time study, first define the population of requests and the sampling procedure. A correctly computed average can still answer the wrong question. Independence, representative sampling and a stable target population are assumptions to examine, not properties created by applying a statistical formula.
Distinguish spread from uncertainty of an estimate
The sample standard deviation s describes the spread of individual observations. Under appropriate independent, identically distributed sampling, s divided by the square root of n estimates the standard error of the sample mean. With n=100 and s=20 milliseconds, that standard error is two milliseconds. A confidence interval uses an appropriate quantile, often from a t distribution; a large-sample normal approximation uses approximately 1.96 for a two-sided 95 percent interval. Autocorrelated measurements do not supply the same information as independent ones, so request count alone cannot justify the usual calculation.
State what a test evaluates
A p-value is computed under a null hypothesis and a specified testing model. It is not the probability that the null hypothesis is true. A statistically significant one-millisecond API improvement might be operationally unimportant, while a nonsignificant difference does not establish equivalence. Report an effect estimate, uncertainty interval and measurement conditions alongside the test. Repeatedly checking results and stopping at a convenient moment, or selecting the smallest p-value from many metrics, changes the interpretation. A decision rule needs to match the actual data-collection and analysis procedure rather than an idealized one used only in the report.
Make the comparison reproducible
When comparing two implementations, record load, warm-up, execution order and error rates. Means can hide a small set of very slow requests, making quantiles useful, but a p99 estimate needs enough tail observations to be stable. Specify an outlier rule before inspecting which implementation benefits from removal. Failure-related delays are often part of the behavior being measured, not disposable noise. Finally, explain which hardware, time period and user population the estimate can reasonably represent. Reproducibility requires both a calculation and the operational context that generated its inputs.