Experiments and implementations · Intermediate

Comparing inference batch size and latency

Use a deterministic queueing model to compare batch setup savings with individual request latency, without claiming GPU benchmark results.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. Define the simulation boundary
  2. Define request latency
  3. Inspect completion and per-request delay
  4. Compare more than one aggregate
  5. Vary arrivals and policy explicitly
  6. Tie the conclusion to an objective
  7. Ask Logeia about this articleCheck your understanding

Define the simulation boundary

Four requests arrive at times zero, two, four, and six milliseconds. One server processes one batch at a time, with invented service time two milliseconds of setup plus one millisecond per request. This is an explanatory function, not measured inference performance. Requests are equal-sized, and transfer, prefill, and decode are collapsed into one service cost. Real length-dependent execution is excluded to isolate batch-formation waiting and setup savings. Clearly identifying these simplifications prevents a numerical queueing exercise from being mistaken for a benchmark of a model, accelerator, or production serving policy.

Define request latency

Compare batch sizes one, two, and four. A batch begins only after all its assigned requests have arrived and the preceding batch has finished. A final incomplete group would be processed at its actual size. Request latency is completion time minus arrival time, including both formation waiting and service. There is no timeout or continuous admission within an active batch; the fixed arrival list is partitioned in order. Save the standard-library-only code as batch.py and run it with Python 3. It creates no service state or external data dependencies.

Inspect completion and per-request delay

Batch size one finishes at 12 ms with latencies[3,4,5,6]. Size two finishes at 10 ms with latencies[6,4,6,4]. Although the overall workload finishes sooner, the first request waits longer for its batch mate. Size four finishes at 12 ms with latencies[12,10,8,6]. Assertions verify these timelines. The final printed rate divides four requests by the finite workload’s completion time; it is not a steady-state capacity estimate or measured tokens per second. Keep both per-request delays and makespan rather than declaring one policy universally faster from a single aggregate.

python
arrivals = [0, 2, 4, 6]
def simulate(size):
    available, latencies = 0, []
    for i in range(0, len(arrivals), size):
        batch = arrivals[i:i+size]
        start = max(available, batch[-1])
        finish = start + 2 + len(batch)
        latencies.extend(finish-a for a in batch)
        available = finish
    return available, latencies
for size in [1, 2, 4]:
    finish, latencies = simulate(size)
    print(size, finish, latencies, sum(latencies)/len(latencies), len(arrivals)/finish)
assert simulate(1) == (12, [3,4,5,6])
assert simulate(2) == (10, [6,4,6,4])
assert simulate(4) == (12, [12,10,8,6])

Compare more than one aggregate

Mean latency is 4.5 ms for size one,5 ms for size two, and 9 ms for size four. Finishing the whole workload earlier and having a worse mean latency are not contradictory: makespan and individual response experience measure different things. Four requests also do not support a stable p 99 estimate. Showing every delay is clearer here. A larger real experiment should include distributions, completion, rejection, and deadline failures. Avoid selecting whichever aggregate favors a proposed change while concealing the user-visible metric that becomes worse under the same policy.

Vary arrivals and policy explicitly

If every request arrives at zero, formation waiting disappears and setup savings become more direct. Larger arrival gaps instead increase the cost of waiting for a group. Adding a wait limit requires a rule for processing a batch before it fills. Continuous batching introduces replacement of completed requests with new ones, a mechanism absent here. When extending the exercise, record the arrival sequence and policy changes without simultaneously altering every service-cost assumption. This makes it possible to explain why the result changed rather than simply generating another favorable timing table.

Tie the conclusion to an objective

Tabulate makespan, mean latency, all individual latencies, and the finite-workload rate for each setting. Under an illustrative goal of returning the first request within 5 ms, identify which policy meets it for the original arrivals. Completion means declaring the optimization objective and separating throughput from waiting. Applying the reasoning to a service requires another experiment recording input and output lengths, cache capacity, hardware, and load. The model’s numbers are neither GPU performance claims nor recommended batch sizes; they expose the tradeoff that an implementation-aware measurement should investigate.

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Prepare for AI systems · STEP 21

Choose a learning path