Define the payload being modeled
This experiment calculates cache tensor payload without allocating GPU memory. With two layers, two KV heads, head dimension four, and two bytes per element, each stored position uses sixty-four bytes. The leading factor two represents keys and values. Two requests currently have lengths one and seventeen. Compare their live positions with positions reserved by different allocation policies. Model weights, temporary workspaces, and page metadata are outside the calculation. Explicitly limiting what the formula counts is necessary before comparing its result with actual device memory usage or making a claim about a model fitting on hardware.
Specify allocation policies
Live payload multiplies eighteen total current positions by sixty-four bytes. Paging rounds each request separately to sixteen-position units, producing sixteen and thirty-two positions. Maximum reservation assigns sixty-four positions to each request immediately. Rounding the combined eighteen only once would model a different allocator. Here unused space at the end of one request’s page is not shared with another request. Prefix sharing and sliding-window eviction are also excluded. These explicit simplifications isolate allocation policy and make the arithmetic auditable rather than implying a universal behavior of all inference servers.
Run the calculator
Save the code as capacity.py and run it with Python 3 and the standard math module. Expected byte counts are 1152 live,3072 paged, and 8192 maximum-reserved. The calculator also reproduces the guide’s twenty-four-layer example as 384 MiB. Assertions verify arithmetic and units; they do not demonstrate a successful allocation on a device. No model download, external data, or service is required, and cleanup concerns only the script. Recompute results from the printed bytes-per-position value so that an incorrect factor or rounding rule can be diagnosed independently of the final totals.
import math
layers, kv_heads, head_dim, element_bytes = 2, 2, 4, 2
lengths, page, maximum = [1, 17], 16, 64
per_token = 2*layers*kv_heads*head_dim*element_bytes
live = sum(lengths)*per_token
paged = sum(math.ceil(t/page)*page for t in lengths)*per_token
reserved = len(lengths)*maximum*per_token
print('bytes per token', per_token)
print('live/paged/max reserved', live, paged, reserved)
assert (live, paged, reserved) == (1152, 3072, 8192)
large = 2*24*8*64*4*2048*2
print('large cache MiB', large/(2**20))
assert large == 402653184
Inspect page-boundary jumps
Reducing the second length from seventeen to sixteen changes paged reservation to thirty-two positions, or 2048 bytes. Crossing from sixteen to seventeen therefore adds one whole 1024-byte page although live payload grows by only sixty-four bytes. This jump follows the allocator’s unit size rather than an arithmetic bug. Repeat with page size eight, rounding requests separately. Smaller pages do not automatically mean faster execution: management and access costs are outside this payload model. The experiment explains capacity fragmentation, not the full performance tradeoff among paging implementations.
Include future growth
Admission based only on current length can exhaust capacity as generation continues. Define whether the maximum sixty-four includes input plus future output, and decide what happens when requests exceed it. Reducing maximum in the calculator requires a validity check as well as a reservation formula. Lower-precision cache storage may reduce element bytes but add scales or metadata. Replacing two bytes by one in arithmetic does not establish that an implementation actually stores or computes in that format. Capacity planning must combine predicted growth, policy limits, and the representation genuinely supported by the serving system.
Report all three quantities
Tabulate the three byte counts for lengths[1,16],[1,17], and[16,32]. Completion means explaining per-request rounding, the key/value factor, and MiB conversion. Do not reinterpret these values as total model memory or concurrency performance. For hardware verification, record model version, actual tensor shapes, dtype, and allocation policy, then observe used and reserved memory separately. Investigate weights, temporary buffers, metadata, and other requests when measurements differ. That separation distinguishes a wrong formula from legitimate allocations outside the modeled cache payload.