Study guides · Intermediate

Inference-service SLOs: separate waiting from generation speed

Define user-facing objectives for first-token delay, generation cadence, completion time, and rejected requests without hiding queueing or failures.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. Define the user-visible interval
  2. Separate indicators, objectives, and agreements
  3. Inspect the tail rather than only the mean
  4. Match admission to service capacity
  5. Evaluate batching as a trade-off
  6. Connect objectives to operating decisions
  7. Ask Logeia about this articleCheck your understanding

Define the user-visible interval

Inference APIs expose several waiting intervals. Time until the first token differs from the time between generated tokens and from total completion time. A fast start can coexist with pauses later, while high generation throughput can follow a long initial wait. Define which experience matters and specify measurement start and end points. GPU execution time alone omits network, authentication, retrieval, and queueing delays experienced by the user. Beginning a stream also differs from successfully completing the request. Metrics must reflect the operation users actually depend on rather than only the most convenient internal component to instrument.

Separate indicators, objectives, and agreements

An SLI is a measured indicator, an SLO is a target, and an SLA adds an agreement and consequences. An illustrative objective might require ninety-five percent of eligible requests to receive an initial output within two seconds over a specified period. Those numbers are not universal recommendations. Eligibility, time window, and measurement location are essential parts of the definition. Short interactive replies and long document generation can require different objectives. Understand the user’s task and the consequences of delay before selecting a convenient percentile or copying another service’s target into a contract.

Inspect the tail rather than only the mean

Nine one-second requests and one eleven-second request have a two-second mean, while one user waits eleven seconds. This tiny sample does not reliably estimate a p99, but it illustrates how means hide variation. Inspect distributions by input length, output length, retrieval use, model, and priority. Mixing different request classes can make a change in traffic composition look like a system regression. Report the aggregation interval and sample size. Do not average separate percentile values and label the result the global percentile: the underlying samples or an appropriate aggregation structure are needed to calculate the combined distribution.

Match admission to service capacity

In a simplified example, arrivals of fifty requests per second with service capacity of forty under unchanged conditions grow backlog by ten requests per second. This is arithmetic, not a measured inference benchmark. Enlarging the queue does not resolve a persistent capacity deficit. Define bounds, deadlines, and overload rejection or deferral. Actual service capacity depends on input and output lengths and concurrency, so a fixed requests-per-second number is incomplete. Track rejected work too: excluding all rejected requests can make the latency of admitted requests look excellent while users experience a deteriorating service. Admission and latency objectives must be evaluated together.

Evaluate batching as a trade-off

Batching can improve accelerator utilization, but waiting to form a batch adds latency and mixed request lengths create scheduling trade-offs. Consider a batching wait limit, continuous scheduling, KV-cache capacity, and cleanup after cancellation rather than only a fixed batch size. Higher token throughput does not establish better interactive experience if initial output becomes slower. Compare first output, generation cadence, and completion success under the same request-length distribution and offered load. The objective determines the acceptable trade-off. Throughput, latency, memory use, and fairness are related constraints, not interchangeable ways of reporting one universal performance score.

Connect objectives to operating decisions

Use an objective to guide operating decisions by tracking violations over its defined window. Per-request timing that separates waiting, retrieval, prefill, and decoding helps locate a change more effectively than an average-only dashboard. Measurement does not require unlimited retention of prompts or sensitive content. Choose minimal identifiers and task-relevant fields such as lengths, durations, and outcomes. Reevaluate under comparable conditions after changing a scheduler or capacity policy. Check that improving one request class has not starved another. An SLO is useful when its definition, observations, and decision rules remain connected, not merely when a favorable number is displayed.

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Prepare for AI systems · STEP 21

Choose a learning path