Experiments and implementations · Intermediate

Comparing chunk boundaries and retrieval results

Compare fixed windows, overlapping windows, and a complete semantic unit on a tiny rule-and-exception document with a top-one retrieval budget.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. Require both a rule and its exception
  2. Fix the retrieval budget
  3. Check an explicit completeness condition
  4. Understand the overlap counterexample
  5. Account for larger-unit costs
  6. Evaluate beyond one hand-built sentence
  7. Ask Logeia about this articleCheck your understanding

Require both a rule and its exception

Use an invented policy: standard cancellations receive a refund within thirty days, except customized orders are never refundable. The question compares standard and customized conditions, so both the deadline and exception are required. A question about only one condition would have different evidence requirements. This is not a real merchant policy. The code uses a short English word sequence split on spaces rather than a model tokenizer. That simple unit exposes boundary effects without adding learned-model behavior, financial advice, or assumptions about the token counts of a deployed retrieval system.

Fix the retrieval budget

Compare eight-word windows, eight-word windows with four-word overlap, and the entire sentence as one semantic unit. Retrieve only one result using overlap with three query words; ties follow original order through Python’s max behavior. This is lexical matching, not an embedding benchmark. Retrieving two pieces and combining them might solve a case that top-one retrieval cannot, but that changes the budget and is a separate experiment. Fixing one retrieved unit makes it easier to inspect whether a boundary separates the two conditions required by this particular comparison question.

Check an explicit completeness condition

Save the code as chunk.py and run it with Python 3. The required words thirty, customized, and never must all occur in the selected piece. This is an explicit check for this tiny sentence’s two conditions, not a general semantic grader. Assertions verify failure for both window policies and success for the whole sentence. No external data, file mutation, or service is involved. Read the selected text as well as the boolean result; a word-set predicate can check this controlled example but cannot establish completeness for arbitrary paraphrases, negation, or real policy documents.

python
words = 'standard cancellations receive a refund within thirty days except customized orders which are never refundable'.split()
query = {'refund', 'thirty', 'customized'}
required = {'thirty', 'customized', 'never'}
def chunks(size, overlap=0):
    assert 0 <= overlap < size
    return [words[i:i+size] for i in range(0, len(words), size-overlap)]
def retrieve(parts):
    return max(parts, key=lambda part: len(set(part) & query))
for name, parts in [('fixed8', chunks(8)), ('overlap8', chunks(8,4)), ('whole', [words])]:
    selected = retrieve(parts)
    complete = required <= set(selected)
    print(name, len(selected), ' '.join(selected), 'complete', complete)
assert not (required <= set(retrieve(chunks(8))))
assert not (required <= set(retrieve(chunks(8,4))))
assert required <= set(retrieve([words]))

Understand the overlap counterexample

Overlap does not guarantee that conditions farther apart than one window appear together in a single retrieved result. Repeating the standard rule across many overlapping candidates also does not guarantee retrieval of its exception. This counterexample does not show that overlap is always useless; it shows insufficiency under the stated width and budget. Try wider windows, structural grouping, or adding a parent section as separate interventions. Record context length after each change so an improvement from better boundaries is not confused with simply allowing more retrieved material or a larger total context budget.

Account for larger-unit costs

The complete sentence contains only fifteen words, so retaining both conditions is cheap here. Treating an entire large document as one chunk can add irrelevant material and cost. This example cannot select one universal chunk size. Consider headings, table headers, footnotes, versions, and the structure connecting rules with exceptions. Do not merge differently authorized sources merely because their text belongs to one semantic topic. Operational comparisons should align deduplication, provenance, and context budgets and retain what was actually passed to the generator, not only the index’s original chunk list.

Evaluate beyond one hand-built sentence

Extend the exercise to width twelve, then separately allow two retrieved pieces to be combined. Tabulate completeness and the number of words passed onward. Completion means defining needed evidence before ranking and avoiding success judgments based solely on a retrieval score. For real documents, include nearby and distant exceptions, tables, and questions requiring multiple sources. Keep tuning questions separate from final assessment. A hand-built sentence explains a failure mechanism; it does not establish aggregate improvement across a corpus, and favorable examples should not be selected as substitutes for a fixed evaluation set.

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Prepare for AI systems · STEP 22

Choose a learning path