Experiments and implementations · Intermediate

Checking confidence-interval coverage through repeated samples

Repeatedly sample a known normal population and compare coverage of the usual interval with a deliberately narrower interval.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. Use a known population
  2. Compare two rules on shared samples
  3. Count coverage events
  4. Quantify Monte Carlo error
  5. Vary sample size separately
  6. Explain the result rather than only printing it
  7. Ask Logeia about this articleCheck your understanding

Use a known population

Generate independent samples of size twenty-five from a normal distribution with known mean three and standard deviation two. Because the population mean is known in this experiment, each interval can be checked against the truth. In an ordinary survey, the unknown mean cannot be inspected that way for the one realized interval. This distinction motivates coverage simulation. Restrict the experiment to a known-variance normal model; do not silently transfer its result to an unknown-variance t procedure or to arbitrary nonnormal populations. Each construction has its own assumptions and interpretation.

Compare two rules on shared samples

Construct two intervals from each shared sample: mean plus or minus 1.96 sigma over square root n, and the same formula with multiplier one. Their half-widths are 0.784 and 0.4. Sharing samples isolates the construction-rule difference rather than introducing unrelated sample variation. Save the standard-library-only code as coverage.py and run it with Python 3. No external dataset or service is needed. Record the seed, population parameters, sample size, and repetition count. The program creates no persistent service state; cleanup only involves the saved script if it is no longer needed.

Count coverage events

For each repetition, draw twenty-five fresh values and compute their mean. Count success when the known population mean lies between the endpoints. Successes divided by two thousand estimate each construction’s coverage. Expect approximately 0.95 for the wider interval and 0.68 for the narrower one. Assertions check only a broad neighborhood and the ordering of coverage. A single seed’s observed proportion is not the exact coverage of the procedure. Independent repetition, a known population, and an explicitly defined inclusion event are what make the estimate interpretable rather than just another percentage.

python
import random, math
rng = random.Random(20261028)
mu, sigma, n, trials = 3.0, 2.0, 25, 2000
hits = {1.0: 0, 1.96: 0}
for _ in range(trials):
    mean = sum(rng.gauss(mu, sigma) for _ in range(n)) / n
    for z in hits:
        half = z * sigma / math.sqrt(n)
        hits[z] += mean - half <= mu <= mean + half
rates = {z: hits[z] / trials for z in hits}
print(rates)
assert 0.92 < rates[1.96] < 0.98
assert rates[1.0] < rates[1.96]
print('half widths', sigma / math.sqrt(n), 1.96 * sigma / math.sqrt(n))

Quantify Monte Carlo error

The Monte Carlo standard error of estimated coverage is approximately square root of p times one minus p divided by repetitions. At p=0.95 and two thousand repetitions it is about 0.0049. Deviations of several thousandths are unsurprising. Quadrupling repetitions halves that simulation uncertainty but does not change the width of any individual interval. Distinguish sample size n from repetition count R: increasing n narrows each interval, whereas increasing R estimates the procedure’s coverage more precisely. Record both so readers can tell which uncertainty a change is intended to reduce.

Vary sample size separately

Change n from twenty-five to one hundred while preserving the normal population and multiplier 1.96. Half-width falls from 0.784 to 0.392, but coverage remains approximately ninety-five percent under this known-variance model. Contrast that with reducing only the multiplier, which narrows the interval and reduces coverage. Experiments with nonnormal distributions are useful extensions, but should follow verification of the baseline calculation. In small samples or heavy-tailed populations, reusing a numerical multiplier does not by itself preserve coverage. Label every output with the sampling assumptions that generated it.

Explain the result rather than only printing it

Report half-width, coverage count, repetition count, and estimated coverage for each multiplier. Explain separately that an interval is narrow and that a construction contains the population mean at a specified long-run frequency. Completion requires distinguishing n from R and explaining why ninety-five percent does not refer to the proportion of individual observations enclosed. Counting sample points inside their own mean interval would measure a different event. Revisit the companion guide’s distinction between a fixed parameter and a random interval if those interpretations become mixed while reading the simulation output.

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Prepare for AI systems · STEP 11

Choose a learning path