Locate the first output
An autoregressive decoder first processes a prompt during prefill, producing past representations and next-token scores. After selecting the first token, it repeatedly processes new positions during decode. Implementations and measurements can label the first step differently, so define operational boundaries. In an explanation where prefill supplies the first selection, returning five tokens involves four subsequent inter-token intervals. Do not automatically multiply a decode duration by the output count without specifying whether the first token is already accounted for. This convention matters when reconciling internal traces with an end-to-end request timeline.
Compare available parallelism
Prefill processes multiple prompt positions together and can exploit matrix-operation parallelism. A causal mask does not require calling the model separately for every prompt position. Ordinary autoregressive decode instead depends on the previously selected token, so all not-yet-generated positions for one request cannot simply be finalized simultaneously. Batching independent requests and speculative techniques are separate mechanisms. Distinguish parallelism within prompt computation, dependency across generated positions, and parallelism across requests. GPU parallel execution alone does not remove the model’s sequential token-selection dependency or guarantee low latency for every phase.
Identify what KV reuse saves
A KV cache stores keys and values for past positions so a new query can reuse them. It avoids recomputing the entire past sequence, but does not eliminate attention to cached keys or the new position’s projections and feed-forward work. Dense prompt attention involves quadratically many position pairs; one new query attends over the existing length. Those terms do not alone predict full-model runtime, which also includes weight reads and other layers. Identify the computation actually removed, the cache capacity required, and the sequence range being reused rather than treating caching as making subsequent generation free.
Reconcile a request timeline
In an invented timeline, queueing takes 20 ms, prefill takes 100 ms including the first selection, and four later intervals take 10 ms each. First output arrives at 120 ms and the fifth token at 160 ms. These are illustrative durations, not hardware measurements, and omit network or retrieval. Adding a50 ms retrieval stage moves first output to 170 ms. A trace showing 100 ms prefill alone cannot explain user-observed time to first token. Align timestamps and account for every stage once. The purpose is to reconcile definitions, not advertise throughput from a simplified arithmetic model.
Measure comparable workloads
Compare matched prompt lengths, output lengths, concurrency, batching policy, model, and precision. A pooled average across short and long prompts can conceal a phase-specific change. Decode intervals may also change as cache length grows. GPU work can be issued asynchronously, so host-call duration is not necessarily execution duration; use suitable synchronization or instrumentation. Record warmup, existing cache state, and cancellation. Inspect distributions and completion rates as well as means. Otherwise a change in workload composition or admission policy can masquerade as a faster prefill or decode implementation.
Choose a phase-specific intervention
If first output is slow, first separate retrieval, queueing, and prefill. If later intervals are slow, inspect decode weight traffic, KV reads, and concurrency effects. Reducing evidence, reusing prefixes, and changing batching each introduce different resource and quality tradeoffs. Extend the illustrative five-token request to ten outputs; under the same convention the final timestamp becomes 210 ms. Completion means explaining first-output timing separately from subsequent intervals and distinguishing a timing model from measured performance. Use the SLO and batching materials to decide which user-visible outcome an optimization should improve.