Experiments and implementations · Intermediate

Exploring estimation variability with bootstrap samples

Resample an observed dataset with replacement to examine mean variability and the limits of information available in the sample.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. Identify the resampling population
  2. Keep each resample size fixed
  3. Inspect standard error and percentiles
  4. Expose missing information
  5. Respect dependence and grouping
  6. Report the procedure and its limits
  7. Ask Logeia about this articleCheck your understanding

Identify the resampling population

Treat the observed values 2,3,4,5,11 as an empirical distribution with equal probability on each observation. Each bootstrap sample draws five values with replacement, so some observations may appear repeatedly and others not at all. Merely shuffling the original values would leave the mean unchanged and would not estimate sampling variability. Bootstrap resampling approximates repeated sampling using the observed dataset; it does not collect new independent evidence from the original population. This distinction is essential when interpreting the apparent abundance of resampled values printed by a simulation.

Keep each resample size fixed

Save the standard-library code as bootstrap.py and execute it with Python 3. It constructs four thousand means from resamples of size five, using a fixed seed without changing the original data. Increasing the repetition count improves the simulation of the empirical distribution but does not increase the original observation count. The standard deviation of resampled means approximates the standard error of the mean under this bootstrap model. No network or output-file state is created. Cleanup only concerns the script, while the original sampling mechanism still needs its own substantive explanation.

Inspect standard error and percentiles

The original sample mean is five. Sort the bootstrap means and form a simple percentile interval using explicitly selected order statistics: the hundredth and thirty-nine-hundredth of four thousand values. This rule can differ from interpolated library quantiles. Treat the resulting interval as a percentile approximation, not a guaranteed exact ninety-five percent confidence interval. The bootstrap standard error is the standard deviation of bootstrap means, not the standard deviation of the original observations. Identify both the statistic being recomputed and the distribution whose spread is being summarized.

python
import random, statistics
rng = random.Random(20261028)
data = [2, 3, 4, 5, 11]
means = [statistics.mean(rng.choices(data, k=len(data))) for _ in range(4000)]
means.sort()
se = statistics.stdev(means)
lo, hi = means[99], means[3899]
print('mean', statistics.mean(data), 'bootstrap SE', round(se, 4))
print('percentile interval', lo, hi)
assert lo <= hi and se > 0
constant = [statistics.mean(rng.choices([5]*5, k=5)) for _ in range(100)]
assert statistics.pstdev(constant) == 0
print('constant sample bootstrap SE', statistics.pstdev(constant))

Expose missing information

Resampling 5,5,5,5,5 always produces mean five and estimated standard error zero. This does not prove that the original population has no variability. The observed sample may have missed alternatives or rounded distinct measurements to one value. Resampling cannot invent tail behavior or distinctions absent from the empirical distribution. Increasing repetitions does not automatically recover unseen extreme observations. Representativeness and measurement resolution remain assumptions of the analysis. The constant-sample comparison is deliberately simple so that the limitation can be understood without confusing it with implementation details or Monte Carlo noise.

Respect dependence and grouping

Ordinary observation-level resampling treats the observations as independent and identically distributed. Repeated measurements from one user or a time series violate that simple structure. Sampling rows independently can destroy dependence and understate uncertainty. Cluster or block resampling instead preserves a chosen dependence unit, but requires a decision about which units can reasonably be treated as independent. Some statistics, including certain boundary or extreme-value quantities, also need special care. The label bootstrap does not validate every resampling scheme; the statistic and collection process determine which approximation is defensible.

Report the procedure and its limits

Compare the original data with a version replacing eleven by seven, then compare repetition counts one thousand and four thousand. Separate changes to observed information from changes to simulation precision. Report seed, original sample size, repetitions, quantile rule, and independence assumptions. Explain the constant-sample zero standard error instead of only listing endpoints. To investigate actual coverage, add an outer simulation that draws fresh original samples from a known population and reruns the bootstrap construction. Repeating resamples from the same fixed dataset alone does not estimate that outer repeated-sampling property.

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Prepare for AI systems · STEP 11

Choose a learning path