Experiments and implementations · Intermediate

Separate first and repeated timing observations

Save first, repeated and empty timing intervals, recompute medians from raw samples, and distinguish observations from unverified causes.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. Define what the first observation means
  2. Mark the measured boundary in the program
  3. Record empty intervals with the same clock
  4. Build summaries without losing original samples
  5. Do not invent causes from a timing difference
  6. Preserve run order alongside sorted summaries
  7. Solve the summary exercise
  8. Ask Logeia about this articleCheck your understanding

Define what the first observation means

Repeated calls can differ because of page use, instruction or data caches, initialization, frequency and scheduling. This experiment saves eleven C array-sum observations, including the first, together with empty timing intervals. Read the memory-hierarchy material first. The objective is an inspectable protocol that does not select a favorable run or assign every initial difference to a single warmup explanation.

The input contains 1,048,576 uint32_t values, all one. Allocation and initialization finish before timing, already touching the array's pages. Observation zero is therefore the first call after that preparation, not an unprepared universally cold execution. This native C program has no JIT, so a difference cannot be attributed to JIT compilation. State the preparation explicitly; another reader should be able to reconstruct what first means without assuming every cache or page began empty.

Mark the measured boundary in the program

Save warm.c and build with gcc -O2 -std=c11 -fno-ipa-pure-const warm.c -o warm, then run it on Linux. CLOCK_MONOTONIC measures elapsed time rather than an adjustable calendar clock. Time the walk call, then assert that its result equals n and consume it through sink. Allocation, initialization, validation and printing remain outside the timed interval.

The extra compiler flag makes the repeated-call study easier to inspect; noinline alone is not a universal guarantee against elimination or movement. Verify the actual call and reads in disassembly. The recorded environment uses GCC 13.3 on a virtualized CPU exposing an EPYC model. A reproduction should also save OS, flags, element count and repetition count. If preceding work materially affects conditions, record it rather than deleting observations afterward to force a preferred pattern. The boundary and preparation are part of the experimental result.

Record empty intervals with the same clock

Each repetition first calls ns twice and prints their difference as empty, then measures walk separately. Empty intervals include clock-call and associated overhead. A zero observation can reflect resolution or call conditions rather than literally costless execution. Compare their scale with walk durations to identify intervals too short for confident interpretation. The code does not subtract one empty sample from every walk observation.

Overhead is not perfectly constant, so naive subtraction can create negative or unstable estimates. Any additional subtraction analysis needs its distribution and error treatment stated. Grouping many calls inside one interval can reduce relative timer cost, but repeated reuse changes cache and state conditions. A single-call latency question differs from a steady-state throughput question. More internal repetitions do not automatically measure the same quantity with greater accuracy. Preserve the actual protocol so a reader can see which question the resulting statistic answers.

c
#define _POSIX_C_SOURCE 200809L
#include <assert.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <time.h>
static volatile uint64_t sink;
static uint64_t ns(void) { struct timespec t; assert(!clock_gettime(CLOCK_MONOTONIC,&t)); return (uint64_t)t.tv_sec*1000000000u+t.tv_nsec; }
__attribute__((noinline)) static uint64_t walk(const uint32_t *a,size_t n) {uint64_t sum=0;for(size_t i=0;i<n;i++)sum+=a[i];return sum;}
int main(void) {
 size_t n=1048576;uint32_t *a=malloc(n*sizeof(*a));assert(a);for(size_t i=0;i<n;i++)a[i]=1;
 for(int r=0;r<11;r++) {uint64_t begin=ns(),end=ns();printf("empty,%d,%llu\n",r,(unsigned long long)(end-begin));
  begin=ns();uint64_t sum=walk(a,n);end=ns();assert(sum==n);sink=sum;printf("walk,%d,%llu\n",r,(unsigned long long)(end-begin));
 }free(a);return 0;
}

Build summaries without losing original samples

A passing run prints eleven empty and eleven walk rows, with every walk result equal to n. Summarize all observations first, then optionally add separate summaries for observation zero and repetitions one through ten. Keep the raw values and explain the exclusion rule. No assertion requires the first call to be slowest; a later sample can be delayed, and position alone does not make a value an outlier.

For the illustrative series 10, 4, 5, 4, 6 ns, the full median is five while the remaining four values have median 4.5. These are teaching numbers, not this program's observed times. They show how reporting rules affect the summary. Quantile definitions also differ for small samples, so record the selected method. Eleven observations do not characterize every long-term machine condition. The immediate goal is a summary another reader can recompute from the retained observations and declared protocol.

Do not invent causes from a timing difference

The recorded run verifies successful code execution, output, interval, durations and generated instructions. PMU counts are unavailable, so first-call cache and translation misses are unmeasured. Initialization already touches pages, making a universal first-call-means-page-fault explanation unjustified. Frequency and scheduling are additional candidates. A causal follow-up needs corresponding counters, traces or controlled changes, saved as additional evidence.

Fresh process launches differ from repeated traversal in one process. Investigating JIT warmup in another language additionally requires compilation or optimization observations; this C timing cannot establish JVM or browser behavior. Application relevance matters too: a one-time user action can depend more on first-use latency than on a best steady-state sample. Choose the protocol according to the decision it should inform, rather than selecting whichever summary makes an implementation look quickest.

Preserve run order alongside sorted summaries

Empty followed by walk is part of this protocol. Measuring every empty interval first and every walk afterward creates another ordering condition. Keep repetition indices even when sorting values to compute a median. The same raw observations can then support a summary and a time-order inspection of drift or isolated pauses. A sorted-only report loses that distinction. Retaining order is inexpensive here and avoids mistaking a changing environment for a stable property of the first versus repeated call.

Solve the summary exercise

For illustrative samples 20, 8, 10, 9, 8, sorted values are 8, 8, 9, 10, 20 and the full median is nine. Excluding the first leaves 8, 8, 9, 10, whose median is 8.5. Reporting only the minimum eight loses its frequency and surrounding variation. Neither summary is uniquely appropriate without the question and reporting rule. Apply the same declared calculation to the actual CSV while preserving every original observation.

Would replacing walk with printing the expected n be a valid benchmark? No: the output would look correct while the reads and additions were absent. That is why output assertions alone cannot validate a performance experiment. At completion save raw samples, aggregation method, compiler settings and relevant disassembly. Storage is freed and no service remains running. When transferring this method to another lab, redefine its preparation and useful-work boundary instead of assuming all repeated loops answer the same question.

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Build your CS foundations · STEP 8

Prepare for AI systems · STEP 5

Choose a learning path