Study guides · Intermediate

Investigating NUMA placement: follow CPUs and memory pages together

Separate initialization, CPU affinity, and memory policy before evaluating a placement change under controlled workload conditions.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. Distance concerns the physical access path
  2. Track which thread initializes the data
  3. Read policy scope and administrative constraints
  4. Compare locality with bandwidth distribution
  5. Do not equate allocation counters with remote loads
  6. Change one condition at a time
  7. Ask Logeia about this articleCheck your understanding

Distance concerns the physical access path

In NUMA systems, access paths from processors to memory are not uniform. A local node and a remote node can offer different latency or bandwidth from a given CPU. This is not a numerical distance between virtual addresses. The same pointer can refer to a different physical access path when execution or page placement changes. Record CPU placement and memory placement separately. Binding a thread to one node does not automatically migrate every previously allocated page there. A useful investigation connects both states to the workload rather than treating affinity as a complete description of locality.

Track which thread initializes the data

Consider a large anonymous array initialized through writes by one thread, then divided among workers on several nodes. Subject to memory policy and available capacity, allocation triggered by those writes can concentrate pages near the initializing thread. Dividing later computation does not necessarily redistribute the pages. Having each worker initialize its own range is a useful alternative to test. However, read-only first accesses, shared mappings, and huge pages can change allocation behavior. The phrase first touch is not a complete allocation specification. Record the mapping type and initialization procedure alongside CPU placement and policy.

Read policy scope and administrative constraints

Linux memory policy includes placement preferences and interleaving across nodes. Scope matters: a task policy and a mapping-specific policy do not necessarily affect the same allocations. Administrative constraints such as cpusets also restrict usable nodes. Installing a policy does not automatically change where all previously faulted pages reside. Separate future allocation policy from explicit or automatic migration of existing pages. Verify effective allowed nodes and observed placement, not only the requested setting. A configuration file saying that a node is preferred is not evidence that every access uses local memory under every capacity condition.

Compare locality with bandwidth distribution

Placing everything near one node is not universally fastest. Many workers can saturate one memory controller even when their accesses are local. Interleaving can distribute bandwidth demand, while increasing remote-access frequency or latency for individual loads. The trade-off depends on whether the workload is latency-sensitive or dominated by large streaming transfers. Shared writes introduce coherence traffic as another factor, so page placement alone may not explain a slowdown. Choose hypotheses based on the actual access pattern. Locality, controller utilization, and coherence are related resource effects rather than interchangeable labels for any multicore performance regression.

Do not equate allocation counters with remote loads

Numastat and process NUMA information help inspect allocations and placement, but their counters are not automatically counts of every remote CPU load. Check units, accumulation interval, and whether the scope is one process or the system. Hardware-event measurements also depend on processor support and permissions. When direct evidence is unavailable, describe a placement hypothesis rather than claiming that remote access is proved. Compare whether observed distribution and elapsed-time changes are consistent with it. Keep hypothetical explanatory quantities separate from measurements actually collected on the machine; a plausible latency number is not an experimental result.

Change one condition at a time

Separate comparisons that change initialization, CPU placement, or memory policy. Keep input constant and record repetitions, page size, load, and warm-up. Migration itself costs time, so a short task may not recover the setup expense of an improved steady-state placement. Verify identical outputs and distinguish startup from repeated execution. This guide describes an investigation method rather than presenting results measured on a particular NUMA machine. Limit conclusions to the tested conditions. A configuration that helps one access pattern, thread count, or data size does not establish a universal placement rule for every application on the same hardware.

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Build your CS foundations · STEP 8

Prepare for AI systems · STEP 5

Choose a learning path