Experiments and implementations · Intermediate

Calculating KV-cache size from model configuration

Calculate live KV payload, page-rounded allocation, and maximum-length reservation from explicit per-request lengths.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. Define the payload being modeled
  2. Specify allocation policies
  3. Run the calculator
  4. Inspect page-boundary jumps
  5. Include future growth
  6. Report all three quantities
  7. Ask Logeia about this articleCheck your understanding

Define the payload being modeled

This experiment calculates cache tensor payload without allocating GPU memory. With two layers, two KV heads, head dimension four, and two bytes per element, each stored position uses sixty-four bytes. The leading factor two represents keys and values. Two requests currently have lengths one and seventeen. Compare their live positions with positions reserved by different allocation policies. Model weights, temporary workspaces, and page metadata are outside the calculation. Explicitly limiting what the formula counts is necessary before comparing its result with actual device memory usage or making a claim about a model fitting on hardware.

Specify allocation policies

Live payload multiplies eighteen total current positions by sixty-four bytes. Paging rounds each request separately to sixteen-position units, producing sixteen and thirty-two positions. Maximum reservation assigns sixty-four positions to each request immediately. Rounding the combined eighteen only once would model a different allocator. Here unused space at the end of one request’s page is not shared with another request. Prefix sharing and sliding-window eviction are also excluded. These explicit simplifications isolate allocation policy and make the arithmetic auditable rather than implying a universal behavior of all inference servers.

Run the calculator

Save the code as capacity.py and run it with Python 3 and the standard math module. Expected byte counts are 1152 live,3072 paged, and 8192 maximum-reserved. The calculator also reproduces the guide’s twenty-four-layer example as 384 MiB. Assertions verify arithmetic and units; they do not demonstrate a successful allocation on a device. No model download, external data, or service is required, and cleanup concerns only the script. Recompute results from the printed bytes-per-position value so that an incorrect factor or rounding rule can be diagnosed independently of the final totals.

python
import math
layers, kv_heads, head_dim, element_bytes = 2, 2, 4, 2
lengths, page, maximum = [1, 17], 16, 64
per_token = 2*layers*kv_heads*head_dim*element_bytes
live = sum(lengths)*per_token
paged = sum(math.ceil(t/page)*page for t in lengths)*per_token
reserved = len(lengths)*maximum*per_token
print('bytes per token', per_token)
print('live/paged/max reserved', live, paged, reserved)
assert (live, paged, reserved) == (1152, 3072, 8192)
large = 2*24*8*64*4*2048*2
print('large cache MiB', large/(2**20))
assert large == 402653184

Inspect page-boundary jumps

Reducing the second length from seventeen to sixteen changes paged reservation to thirty-two positions, or 2048 bytes. Crossing from sixteen to seventeen therefore adds one whole 1024-byte page although live payload grows by only sixty-four bytes. This jump follows the allocator’s unit size rather than an arithmetic bug. Repeat with page size eight, rounding requests separately. Smaller pages do not automatically mean faster execution: management and access costs are outside this payload model. The experiment explains capacity fragmentation, not the full performance tradeoff among paging implementations.

Include future growth

Admission based only on current length can exhaust capacity as generation continues. Define whether the maximum sixty-four includes input plus future output, and decide what happens when requests exceed it. Reducing maximum in the calculator requires a validity check as well as a reservation formula. Lower-precision cache storage may reduce element bytes but add scales or metadata. Replacing two bytes by one in arithmetic does not establish that an implementation actually stores or computes in that format. Capacity planning must combine predicted growth, policy limits, and the representation genuinely supported by the serving system.

Report all three quantities

Tabulate the three byte counts for lengths[1,16],[1,17], and[16,32]. Completion means explaining per-request rounding, the key/value factor, and MiB conversion. Do not reinterpret these values as total model memory or concurrency performance. For hardware verification, record model version, actual tensor shapes, dtype, and allocation policy, then observe used and reserved memory separately. Investigate weights, temporary buffers, metadata, and other requests when measurements differ. That separation distinguishes a wrong formula from legitimate allocations outside the modeled cache payload.

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Prepare for AI systems · STEP 20

Choose a learning path