Study guides · Advanced

GPU Execution, Memory Bandwidth, Batching, and Parallelism

Estimate GPU work using both operations and bytes, and compare batch size, queueing delay and memory demand with explicit assumptions.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. Choose the boundary being optimized
  2. Compute separate bandwidth and arithmetic bounds
  3. Reuse changes the work per request
  4. Account for waiting to form a batch
  5. Manage sequence length as well as request count
  6. Separate prefill, decode and cache capacity
  7. Measure asynchronous execution deliberately
  8. Decide using throughput and tail behavior
  9. Ask Logeia about this articleCheck your understanding

Choose the boundary being optimized

Divide the path from request arrival to returned result into queueing, CPU preparation, device transfer, accelerator execution, return transfer and postprocessing. Faster device computation can have little end-to-end effect if transfer or queueing dominates. Conversely, adding the durations of overlapping stages can double-count elapsed time. Before comparing configurations, define the start and end events and where execution is synchronized. A device-kernel duration and a client-observed response duration answer different questions. Both can be useful, but one should not be substituted for the other when claiming that a user-facing operation has become faster.

Compute separate bandwidth and arithmetic bounds

For an illustrative operation, suppose eight decimal gigabytes must be read from device memory at an assumed effective bandwidth of four hundred gigabytes per second. Reading alone requires twenty milliseconds. A separate five-millisecond arithmetic lower bound does not imply a five-millisecond operation. Even ideal overlap cannot beat the dominant bound, and non-overlapping work or launch overhead increases elapsed time further. These are assumptions for a budget calculation, not measurements of a named accelerator. Keep decimal gigabytes distinct from binary gibibytes and convert seconds to milliseconds explicitly. Unit consistency matters as much as the division itself.

Reuse changes the work per request

Reusing a loaded weight matrix across several inputs can reduce weight traffic per input. In an intentionally idealized example, eight gigabytes shared by eight requests represent one gigabyte of weight reading per request instead of eight. That does not divide all memory traffic by eight: inputs, outputs and intermediate states still require work. Cache capacity and execution layout determine whether the proposed reuse actually happens. Arithmetic intensity, operations divided by transferred bytes, helps reason about this balance. Specify which memory boundary the bytes cross, because host-device transfer and traffic within device memory are different constraints with different bandwidths.

Account for waiting to form a batch

Batching groups inputs so preparation and data reuse can be shared. A deliberately illustrative rule might execute at sixteen requests or after ten milliseconds. At low load, the first request can consequently wait about ten additional milliseconds before execution. Lower device work per request does not guarantee lower response latency once this waiting is included. Design batch size, maximum collection delay, concurrent batches and queue capacity together. Also specify when expired requests are removed and which new requests are rejected under overload. An unlimited queue can conceal a capacity shortfall while making accepted requests progressively less useful to their callers.

Manage sequence length as well as request count

Requests with different lengths can require very different work even when a batch contains the same number of items. Padding sequences of lengths ten and one thousand to the larger length can waste substantial processing in some implementations. Length grouping or token-based capacity limits may help, but excessive grouping can increase the wait for each group. Implementations that reduce padding do not eliminate every other overhead. Record request count, input tokens and growing output lengths. A shared value called “batch size” does not establish equal workloads across two experiments unless these additional dimensions and the scheduling policy are also comparable.

Separate prefill, decode and cache capacity

LLM prefill processes an available input, while decode repeatedly selects another token. Their computation patterns differ. Advancing several requests together may improve weight reuse, but each request also requires storage for cached Keys and Values. A scheduler that admits or retires requests during generation changes latency according to the boundaries at which those changes are permitted. Fitting model weights into memory is therefore not equivalent to fitting the target concurrency. Budget request state, workspaces and implementation overhead, then identify the costs of eviction, transfer or recomputation if capacity is exceeded. Those mechanisms can alter latency even when the arithmetic work looks similar.

Measure asynchronous execution deliberately

If accelerator submission is asynchronous, timing only the host function call until it returns does not measure device completion. Use the implementation’s synchronization or event facilities and describe what the measurement waits for. First-use preparation, warmup, compilation and allocation can change results, so distinguish initial behavior from steady state. Measurement synchronization can itself disrupt overlap present in normal execution. This section describes measurement design rather than claiming a GPU experiment was performed. A reproducible report should state hardware, software version, numerical precision, relevant power configuration and input distribution, together with the timing boundary.

Decide using throughput and tail behavior

Evaluate completed requests per second alongside queueing, first-output time, completion time, errors and rejection. If throughput plateaus while the queue grows, increasing the batch alone may not resolve overload. Identical mean latency can hide a small set of long requests delaying shorter ones. Separate controlled equal-length comparisons from workloads with realistic arrival and length distributions. Check answer quality or numerical error at the same time as speed. A configuration that finishes more work but violates the response deadline or changes the required result is not interchangeable with the original service merely because its throughput number is larger.

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Prepare for AI systems · STEP 21

Choose a learning path