Concepts and theorems · Intermediate

Context windows

Calculate input-output sequence budgets and distinguish configured length, truncation, retrieval, and effective use of long context.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. Identify what the limit counts
  2. Reserve generation capacity
  3. Treat truncation as a semantic change
  4. Separate acceptance from effective use
  5. Account for resource costs
  6. Evaluate selection within the budget
  7. Ask Logeia about this articleCheck your understanding

Identify what the limit counts

A context window constrains the token sequence a model can handle within a processing setting. Decoder-only generation commonly shares a budget between input and generated output, whereas encoder-decoder systems may impose separate source and target constraints. Identify what is counted and where rejection or truncation occurs before comparing advertised numbers. Character counts and document counts are different units. A serving system may also enforce a smaller request limit than the model configuration. Distinguish architectural capacity, configuration, and service policy rather than treating one large number as a complete operational specification.

Reserve generation capacity

For an illustrative shared limit 4096, allocate 256 tokens to system instructions,128 to the question,2000 to documents, and 112 to formatting. Total input is 2496, leaving 1600 for output. Reserving a maximum output 1024 instead allows 2576 document tokens. These are invented configuration values, not universal model limits. Filling the entire budget with retrieved material can leave no room for generation or cause rejection. Verify the final formatted IDs, including chat-template additions, rather than estimating only the visible document body and assuming the generation service will resolve the conflict automatically.

Treat truncation as a semantic change

Truncation makes a sequence fit but does not ensure retention of necessary assumptions or exceptions. Removing the beginning can discard context, while removing the end can discard the actual question or conclusion. Paired-document inputs also require a policy for which sequence loses tokens. Record lengths and the functional role of removed material. Do not solve observability by logging unrestricted sensitive text; use appropriate identifiers and spans. A successful length check proves compatibility with a budget, not preservation of the information needed to answer the question correctly.

Separate acceptance from effective use

Accepting a long input does not prove equally effective use of every included fact. Position, repetition, irrelevant passages, and contradictory material can influence answers. Evaluate by moving required evidence between the beginning, middle, and end; adding distractors; and asking questions requiring multiple pieces of support. Increasing a configured limit does not guarantee higher answer quality. Check whether cited passages actually contain the support and whether required exceptions survive. Request acceptance is an infrastructure outcome; correctly integrating evidence is a separate task outcome that needs its own evaluation.

Account for resource costs

In ordinary dense self-attention, the number of position pairs grows quadratically with sequence length, although optimized implementations need not materialize the entire quadratic matrix. During generation, KV-cache capacity grows with stored positions and other architectural dimensions. Longer context therefore affects latency, memory, and concurrent-request capacity. Measure prefill time, decode intervals, and cache allocation separately when changing limits. Distinguish reserving every request’s maximum length from allocating according to actual usage. A length budget is both an information-selection problem and a resource-management decision, not merely an interface setting.

Evaluate selection within the budget

Retrieval, deduplication, or summarization can conserve context, but introduce retrieval misses or loss of exceptions. For the 4096-budget example, calculate remaining output after using 2500 and 3000 document tokens. Then require output 1024 and justify which evidence to retain. Completion requires explaining information preservation as well as arithmetic feasibility. Use the retrieval-evaluation and chunking experiments to compare methods on fixed questions. Budget optimization should not select only a shorter prompt; it should identify a reproducible tradeoff between resource use and the ability to support the requested answer.

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Prepare for AI systems · STEP 19

Choose a learning path