Study guides · Intermediate

Evaluating retrieval quality in RAG

Fix relevance judgments and evaluation units, aggregate precision and recall, and diagnose retrieval separately from generation.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. Define evidence before scoring answers
  2. Choose the retrieval unit
  3. Calculate two metrics
  4. Distinguish macro and pooled recall
  5. Protect the evaluation set
  6. Choose changes from diagnosed failures
  7. Ask Logeia about this articleCheck your understanding

Define evidence before scoring answers

RAG retrieves evidence and then asks a generator to use it. A final answer alone does not identify whether retrieval failed or the generator mishandled available support. Prepare evaluation questions and the documents or spans needed to answer them before comparing systems. A passage containing the answer string is not necessarily sufficient evidence when conditions or exceptions matter. Include document version, time, and user permissions in relevance judgments. Define the information legitimately available for retrieval so evaluation does not reward access to unavailable, outdated, or unauthorized material.

Choose the retrieval unit

If document A is split into ten chunks, retrieving five chunks from A does not mean five distinct relevant documents were found. Document-level evaluation deduplicates IDs. Conversely, finding A’s irrelevant introduction may not retrieve the evidence located later in A. Specify whether relevance is judged at document, chunk, or evidence-span level. When chunking changes, map results back to comparable documents or spans. Otherwise a changed number of chunks can alter metrics without improving the information available to answer the question. The unit of counting is part of the experiment, not an incidental indexing detail.

Calculate two metrics

For relevance set{A,B} and top-two results{A,C}, one relevant item is retrieved. Recall at two is one half, as is precision at two. Recall divides by relevant items; precision divides by retrieved items. Extending the prefix of one fixed ranking cannot reduce recall, but may introduce irrelevant passages, lower precision, and consume context. State the fixed-ranking condition. A different retrieval method produces another ordering, so monotonic prefix reasoning does not prove that one method’s recall must exceed another’s merely because its chosen k is larger.

Distinguish macro and pooled recall

Suppose question one retrieves one of two relevant documents and question two retrieves its sole relevant document. Mean per-question recall is 0.75, whereas pooling relevant counts gives two thirds. The aggregation rules weight questions differently. Inspect small subgroups and questions with many required documents. An empty relevance set makes ordinary recall undefined, so declare a policy such as separate no-answer evaluation or explicit exclusion with counts. Do not silently remove difficult questions to improve the score. Reporting both denominators and aggregation rules is necessary for a reproducible comparison.

Protect the evaluation set

Separate questions used to tune embeddings, distance, chunking, deduplication, or reranking from those reserved for final assessment. Repeatedly choosing settings on the same evaluation questions can overfit that set. Revisit judgments when documents or intended users change. Track latency, context length, permission failures, and empty results alongside relevance. Retrieving unauthorized material does not count as successful retrieval merely because it contains a relevant answer. The operational constraints belong to the target task, so they cannot be discarded when optimizing a convenient relevance metric.

Choose changes from diagnosed failures

When evidence is missing, inspect indexing completeness, separation of conditions during chunking, representation, and reranking. If all required support is present but generation fails, retrieving more material may not address the cause. Build a table for two fixed questions containing judgments, result lists, precision, recall, and the evidence actually passed onward. Completion requires explaining denominators and evaluation units and separating retrieval success from answer success. The vector-recall and chunk-boundary labs provide controlled examples for checking this process before drawing conclusions about a deployed RAG system.

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Prepare for AI systems · STEP 22

Choose a learning path