Study guides · Intermediate

Pretraining, Fine-Tuning, Generation, and the KV Cache

Separate pretraining, fine-tuning and generation, then estimate the key/value cache for autoregressive decoding.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. Learning to predict a following token
  2. Fine-tuning changes parameters
  3. Context is not an online weight update
  4. Measure prefill and decode separately
  5. Calculate a cache estimate
  6. A cache formula is not a complete device budget
  7. Sampling settings do not establish truth
  8. Check unequal request lengths
  9. Ask Logeia about this articleCheck your understanding

Learning to predict a following token

Autoregressive pretraining can optimize a distribution for each next token conditioned on preceding tokens. In a short factual sentence, the training target may correspond to part or all of a following word, depending on tokenization. The update increases the probability assigned to the observed target under the chosen objective. This is a statistical prediction task, so incorrect or outdated statements in the corpus can also influence learned parameters. A lower pretraining loss does not certify the truth of a particular generated answer. Keep the mathematical objective separate from downstream evaluation of factuality, instruction following and suitability for a specific application.

Fine-tuning changes parameters

Fine-tuning applies additional data or objectives to update parameters. A method may change all weights, only a subset, or additional small matrices. Teaching an output format and providing access to a newly revised document are different goals with different evaluation needs. Repeating the same example can improve imitation of that example without establishing correctness on unfamiliar questions. Separate tuning data from the final evaluation and include tasks outside the narrow training examples. Also retain checks for previously supported behavior, because a useful local improvement may coexist with forgetting or regressions elsewhere. Parameter efficiency is not the same property as reliable generalization.

Context is not an online weight update

Adding reference material to a prompt normally changes the conditioning input rather than updating model weights during that request. In a retrieval-augmented application, a retriever selects the material used to build this context. More material also increases token count, processing work and cache demand. Updating a search index does not establish that the revised document was actually retrieved for a particular question. Record the question, selected document identifiers, model version and generation settings when investigating a changed answer. These observations help separate effects of changed parameters from effects of changed context, without assuming that every conversation is a new training event.

Measure prefill and decode separately

Prefill processes an already available prompt. Decode repeatedly selects a following token using the accumulated state and feeds the selected token into the next step. A short time to first token does not guarantee a short time to complete the entire answer. Conversely, high token throughput after a long initial wait may still provide a poor interactive experience. Specify prompt length, generated length, concurrency and queueing when comparing services. Separate the initial response delay from intervals between generated tokens and from total completion time. Otherwise, a configuration can appear faster merely because it generates a shorter answer or measures a different portion of the request lifecycle.

Calculate a cache estimate

A causal Transformer can reuse Keys and Values associated with past tokens instead of recomputing their projections at every generation step. A simplified cache estimate is 2LTHdb bytes, where L is layer count, T retained tokens, H the number of KV heads, d head dimension and b bytes per element. With L=2, T=100, H=4, d=8 and b=2, the estimate is 25,600 bytes. Four equally sized requests require 102,400 bytes under those assumptions. The factor two accounts for Keys and Values. Use the KV-head count rather than automatically substituting the Query-head count in architectures where they differ.

A cache formula is not a complete device budget

This estimate covers cached Keys and Values only. It excludes model weights, temporary workspaces, implementation metadata and fragmentation. Requests with different lengths also interact differently with fixed maximum-size reservations and incremental allocation schemes. A shared-prefix optimization can reuse only states compatible with the relevant tokens, model and configuration; semantically similar text does not automatically justify sharing cached tensors. Cross-user reuse additionally needs an explicit confidentiality, authorization and retention policy. A cache fitting in memory establishes neither the safety of sharing it nor the throughput a service can sustain. Treat memory capacity and protection of request-specific state as separate design questions.

Sampling settings do not establish truth

Temperature and top-p adjust selection from a predicted distribution. Lowering temperature does not create evidence for an answer absent from the available information. Interfaces can also differ in parameter ranges and treatment of randomness, even when configuration names look similar. Preserve the model and settings for comparisons and repeat cases when variability matters. Include grounded questions, missing information, long contexts and unauthorized requests in evaluation. After a throughput optimization, reevaluate answer quality and appropriate abstention rather than assuming that unchanged model weights imply unchanged service behavior. The full pipeline, not a single sampling parameter, determines the observed result.

Check unequal request lengths

With the earlier toy parameters, the factor 2LHdb is 256 bytes per token. Two requests retaining one hundred and two hundred tokens therefore require 76,800 bytes of payload cache. Reserving two hundred token positions for both requests instead requires 102,400 bytes, a difference of 25,600. This compares allocation assumptions, not measured waste in a particular implementation. Generated tokens also increase retained lengths, so an admission budget based only on prompt size can understate the later peak. Recompute the bound using the allowed output length and the service’s actual reservation policy.

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Prepare for AI systems · STEP 20

Choose a learning path