Ask whose resource, which interval and what quantity
A service can show low CPU utilization while users wait a long time. Before adding CPU capacity, identify whose resource is measured, over which interval, and in what units. CPU time, elapsed time, resident memory, transferred bytes and waiting time answer different questions. Descriptions such as high usage are insufficient to select a change.
The example runs two short child programs and compares differences of cumulative child CPU counters with elapsed time observed by the parent. It also prints a peak-memory field without subtracting peaks. This distinguishes cumulative counters, current quantities and historical maxima. Measurements vary by environment, so the tests do not require one case to be a fixed multiple of the other. The goal is interpreting compatible quantities rather than manufacturing a performance ranking.
Align interval boundaries before comparing times
The parent measures monotonic elapsed time from before spawning a child until waiting completes, including launch and collection overhead. Child CPU counters separately report user and kernel execution time. Sleeping is not simply added as computation. For parallel work across several CPUs, summed CPU time can exceed wall elapsed time, unlike a naive one-clock interpretation.
CPU percentages also use different denominators: one core, the whole host or an allocated quota can each represent one hundred percent. Host spare capacity does not imply spare quota inside a constrained container. Read the tool's normalization rule. Compare interval length, units, included threads and included descendants before using a reported percentage to conclude that work is computationally expensive or that resources are idle for the process that needs them.
| Metric | Meaning | Distinct from |
|---|---|---|
| CPU time | Accumulated CPU use in the scope | Wall time |
| Wall time | Elapsed interval | CPU occupancy alone |
| Peak RSS | Maximum residency in the scope | Current or cumulative allocation |
| Parent/child values | API-defined observation scope | An automatically additive total |
Subtract counters, not historical peaks
RUSAGE_CHILDREN user and system times accumulate across previously reaped children. Subtract before and after values to isolate the new work in this sequential example. Treating the second final total as the second child's own usage would count the first child too. A server with unrelated children completing during the same interval needs a more precise attribution strategy.
A historical maximum such as ru_maxrss is not additive in the same way. After a large earlier child, a smaller later child may leave the maximum unchanged. Subtracting those peaks does not yield the later child's resident memory. Current-RSS APIs and a child's own peak measurement answer different questions. The output deliberately labels a children peak, preserving the distinction between a counter delta, present quantity and maximum over prior activity.
Run one compute case and one waiting case
One child computes a sum of integer squares and another sleeps for0.05seconds. Sleeping deliberately creates waiting; it is not used to guess when another process is ready. subprocess.run waits for actual child completion. The parent reports monotonic elapsed time and cumulative child CPU deltas, asserting nonnegative values. Even the sleeping child incurs startup and interpreter CPU work, so zero CPU consumption is not required.
The code labels peak RSS in KiB for Linux; a port must check its platform's units. Preserve Python version, OS and complete output. These fields do not by themselves classify every scheduler, storage or lock wait. They establish the more limited distinction between elapsed and CPU time and demonstrate taking compatible differences over a controlled sequential-child interval.
import json,resource,subprocess,sys,time
programs={'cpu':'x=sum(i*i for i in range(2000000));assert x>0','wait':'import time;time.sleep(0.05)'}
for name,program in programs.items():
before=resource.getrusage(resource.RUSAGE_CHILDREN);start=time.perf_counter()
subprocess.run([sys.executable,'-c',program],check=True)
elapsed=time.perf_counter()-start;after=resource.getrusage(resource.RUSAGE_CHILDREN)
result={'case':name,'wall_seconds':elapsed,'child_user_delta':after.ru_utime-before.ru_utime,'child_system_delta':after.ru_stime-before.ru_stime,'children_peak_rss_KiB':after.ru_maxrss}
assert elapsed>0 and result['child_user_delta']>=0 and result['child_system_delta']>=0
print(json.dumps(result))
print('child CPU counters are cumulative; peak RSS is not subtracted')
Account for sharing and nested scopes
Summed process RSS can count shared physical pages more than once, while virtual allocation need not all be resident. Host memory including file cache is not identical to one process's anonymous pages. A large value alone does not diagnose a leak. Inspect object lifetime, sharing and the metric definition before interpreting the number as retained private application data.
Cgroup accounting introduces its own charging scope. Adding host, group, parent and child values can double-count nested quantities; examining one process can also omit descendant or kernel costs. Label the hierarchy and resource categories in a comparison table. Different accounting boundaries may all be useful, but they cannot be combined arithmetically without first identifying overlap and attribution rules. This is a scope question, not a disagreement between necessarily incorrect tools.
Do not infer storage activity from application byte counts
Bytes requested from read, bytes actually returned and physical storage traffic can differ because of short reads, cache hits and readahead. A write returning also differs from satisfying a durability requirement. Connect those events to flush and fsync rather than declaring persistent completion from a byte counter alone.
Aggregate device activity does not necessarily identify one request's delay. Where possible, associate request identity, start, completion and the resource being waited on. Counter differences describe activity volume; tracing which request waited where is another observation. The current CPU/RSS experiment does not benchmark storage bandwidth, so it provides no evidence for a device-performance ranking. Use each metric to answer the question its boundary actually supports.
Detect resets and changes of measured identity
Long-running observation must not subtract counters across a process restart or reset as if one cumulative history continued. Reused PID numbers can also identify a different process. Preserve start time or another identity boundary. A negative delta is a reason to inspect reset or target change before simply clamping it to zero and treating it as a valid sample.
The short parent program limits this issue by collecting children sequentially within one execution. A monitoring service adds interval jitter and missing samples. Converting CPU increments into rates requires the actual measured interval, not only a configured nominal polling period. Save cumulative values, subject lifetime and observation times together. Only then can a series of differences be interpreted as activity by one consistent target rather than arithmetic across unrelated histories.
Include the work introduced by measurement
Child CPU time includes interpreter startup and shutdown, not only the sum-of-squares loop. Setup can dominate a tiny task. Decide whether the question concerns the complete launch-and-run request or computation inside an already running runtime. The same recorded numbers can be appropriate for one boundary and unsuitable for another.
Instrumentation also adds output and synchronization work. Moving printing outside a measured interval or running an empty baseline can clarify scope, but subtracting a baseline does not universally isolate every cost. State those conditions rather than presenting a corrected value as exact by definition. This experiment teaches resource-accounting interpretation, not calibrated arithmetic throughput. Accurate field names and inclusion boundaries make the result usable even when its intentionally broad measurement includes several kinds of work.
Exercise: choose quantities for subtraction
If cumulative child user time increases from2.0to2.3seconds and system time from0.4to0.5, the interval CPU total is0.4seconds. A one-second elapsed interval is not contradictory. The remainder is not automatically disk wait; launching, scheduling and deliberate sleeping can also contribute. Additional evidence is needed to assign causes.
If historical child peak RSS remains30,000KiB before and after, the new child's memory is not therefore zero. It may simply have stayed below a previous maximum. Also explain why adding a parent's current memory to a containing group's current memory may double-count. For every output, identify target, interval, unit and whether it is cumulative, current or a peak. Those four labels make resource accounting interpretable.