Study guides · Intermediate

GPU memory bandwidth and arithmetic intensity

Calculate compute and traffic lower bounds with explicit units and distinguish reuse, batching, and measurement from peak hardware limits.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. Choose a memory boundary
  2. Calculate two lower bounds
  3. Interpret arithmetic intensity
  4. Reason about reuse
  5. Measure representation changes
  6. Turn a bound into a testable hypothesis
  7. Ask Logeia about this articleCheck your understanding

Choose a memory boundary

GPU performance depends on data movement as well as arithmetic. Specify the boundary: traffic from device memory toward compute differs from host-to-device transfer. Arithmetic intensity I is operation count C divided by bytes M crossing that chosen boundary. Cache behavior and reuse can change M for the same mathematical computation. Source-array size is not automatically measured traffic. Define the operation-count convention, byte units, and timed region before calculating intensity. Without those choices, two apparently conflicting performance estimates may simply describe different boundaries or different amounts of reused data.

Calculate two lower bounds

Let effective compute capability be P operations per second and bandwidth W bytes per second. Compute and traffic lower bounds are C/P and M/W; overall time must be at least their maximum in the idealized model. With one billion operations, two hundred million bytes, one trillion operations per second, and one hundred billion bytes per second, the bounds are one and two milliseconds. The lower bound is two milliseconds. These invented inputs do not report achieved hardware performance. Nor does the model require summing the two bounds to three milliseconds when overlap is possible.

Interpret arithmetic intensity

The example’s intensity is five operations per byte. The ratio P/W is ten operations per byte, so traffic imposes the tighter bound at intensity five. Roofline reasoning expresses the corresponding throughput ceiling as the smaller of P and W times I. An implementation need not reach a ceiling built from peak values. Instruction type, parallelism, alignment, and utilization affect effective capability. Calling a workload bandwidth-bound is a statement about specified conditions and a model, not a guarantee that every sequence length or batch size shares the same bottleneck.

Reason about reuse

Reusing one weight matrix across several inputs can reduce bytes per request and increase arithmetic intensity, which is one reason batching can help. Larger batches also consume activation and KV memory and may add queueing, so weight reuse alone does not justify unbounded batching. Matrix shapes and reuse opportunities differ between prompt prefill and short decode steps. Compare matched input distributions and separate individual latency from aggregate throughput. Equal total operation counts do not imply equal runtimes when data movement differs, and a more efficient kernel can still worsen user latency if batch formation waits too long.

Measure representation changes

A smaller weight representation can reduce traffic, but reconstruction or unsupported operations can introduce other costs. Simulating integer levels in Python does not demonstrate compressed device reads. Record dtype, kernel, synchronization, transfer inclusion, and warmup for actual measurements. Other workload or device conditions can influence timing, so one run does not establish a general speedup. Allocated memory, bytes transferred, and elapsed time are related but distinct metrics. A change that reduces one can leave another unchanged or worse, which is why representation claims need an implementation-aware performance evaluation.

Turn a bound into a testable hypothesis

Hold operations fixed and halve traffic: the example’s ideal lower bound falls from two milliseconds to one. Then double only compute capability with the original traffic; the bound remains two milliseconds. Explain which resource changed and why it does or does not matter under these conditions. Completion means treating the calculation as a testable hypothesis rather than a benchmark result. On hardware, measure matched before-and-after workloads and investigate mismatches in traffic, shape, or implementation when observed timing differs from the simple model’s prediction.

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Build your CS foundations · STEP 8

Prepare for AI systems · STEP 5

Choose a learning path