Study guides · Intermediate

Calculating KV-cache capacity

Calculate cache capacity with KV heads and allocation policy, distinguishing used positions, reservations, and total model-serving memory.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. Enumerate the stored dimensions
  2. Use KV heads rather than query heads
  3. Compute bytes and MiB
  4. Separate live usage and reserved capacity
  5. Plan growth and distribution
  6. Validate arithmetic before hardware claims
  7. Ask Logeia about this articleCheck your understanding

Enumerate the stored dimensions

For ordinary per-layer key and value storage, element count is two times layer count L, KV-head count Hkv, head dimension d, and total stored positions. Multiply by bytes b per element for capacity. Equal-length requests give total positions B times T; unequal lengths require summing their lengths. The factor two represents keys and values, not batch size. This formula counts cache tensor payload only. Model weights and temporary computation buffers are separate allocations and must not be silently treated as already included in the result.

Use KV heads rather than query heads

Some multi-head architectures use equal query and KV-head counts; grouped-query or multi-query variants share KV information across query heads. With thirty-two query heads and eight KV heads, use eight in the cache formula. Substituting thirty-two would overestimate the same payload fourfold. Sharing is architecture-dependent, so inspect configuration and stored tensor shape. Do not assume that dividing hidden dimension by query-head count universally determines every implementation’s head dimension. Capacity estimation should follow the tensors actually stored, not a familiar architecture label or an unrelated parameter count.

Compute bytes and MiB

With L24, Hkv8, d64, batch four, length 2048, and two bytes per element, payload is 402653184 bytes, or 384 MiB using powers of two. Decimal megabytes would produce a different number, so label units. Doubling batch or doubling sequence length separately doubles this payload to 768 MiB. These are arithmetic estimates, not measured successful allocations. A different model’s layers, head counts, or dtype require recalculation. The worked example is useful because every factor is explicit and can be changed independently, making a surprisingly large estimate traceable to a particular stored dimension.

Separate live usage and reserved capacity

Two requests of lengths one and seventeen contain eighteen live positions. With sixteen-position allocation pages, they reserve sixteen plus thirty-two, or forty-eight positions. A policy reserving sixty-four positions per request may reserve one hundred twenty-eight. Prefix sharing, sliding windows, and paging change stored or allocated position counts, while metadata and fragmentation add overhead. Distinguish live tensor payload, policy-level reservation, and measured device usage. Reporting only the smallest number can conceal why a service runs out of memory even when the mathematical cache formula appears comfortably below its budget.

Plan growth and distribution

A request’s cache can grow after admission as output tokens are generated. Decide whether to reserve maximum output capacity or allocate incrementally, and define behavior when the budget is exhausted. Multi-GPU distribution does not automatically divide cache by device count: KV-head partitioning, replication, and communication strategies matter. Available cache space is what remains after weights, workspaces, communication buffers, and other requests. Admission control and observed headroom are therefore needed alongside the formula. A payload estimate alone cannot guarantee that a given batch and generation limit will fit safely on a device.

Validate arithmetic before hardware claims

The companion lab computes live usage, paged reservation, and maximum-length reservation for a small configuration without a GPU or model download. Recalculate the 384 MiB example and explain why incorrectly using thirty-two query heads gives 1536 MiB. Completion requires identifying every dimension and unit and avoiding the claim that cache payload equals total GPU memory needed by the model. On hardware, record the actual configuration and allocation policy and reconcile observed usage. First-token latency and generation throughput remain separate measurements; fitting a larger cache does not establish a performance improvement by itself.

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Prepare for AI systems · STEP 20

Choose a learning path