Separate retrieval from generation
A retrieval-augmented system obtains material relevant to a question and gives that material to a generator as context. This can separate document updates from model retraining when an application needs current policies. Successful retrieval, however, is not identical to a correct answer. A refund question can retrieve an obsolete policy, a rule for another contract or a passage missing an exception. Evaluate whether the selected material contains sufficient evidence and whether the generator uses that evidence correctly. A displayed citation is an inspectable pointer, not an automatic certificate that the associated claim is supported.
Keep rules and their exceptions together
Consider a policy whose main paragraph says “within thirty days of purchase” and whose next paragraph excludes business contracts. Passing only the first fragment can produce an incorrect answer to a business-customer question. Preserve section names, document identity, revision date, applicable contract type and useful neighboring paragraphs so that qualifications can be retrieved together. Overlap can reduce boundary loss but can also fill top-ranked results with near-duplicate passages. Neither the smallest possible fragment nor the largest possible fragment is universally preferable. Choose boundaries using the amount of evidence needed to answer realistic questions and check the resulting omissions explicitly.
Combine semantic and exact-match evidence
Embedding retrieval uses vector distance or similarity to identify candidates. It can help with paraphrases while still confusing related documents when an exact product identifier, date or error code is decisive. For a question about product A-12, a semantically similar A-21 manual may be unusable. Consider lexical retrieval and metadata restrictions alongside vector candidates. Scores from different retrieval methods need not share a meaningful numerical scale. Specify a merging or ranking policy, then inspect both candidate coverage and retention after reranking. An answer document absent from the initial candidate set cannot be rescued by a later ranking stage alone.
Enforce access before providing model context
Similarity does not express whether the current user is allowed to read a document. Giving a restricted passage to a generator and removing secrets only from the displayed answer has already disclosed the protected text to that processing stage. Apply compatible authorization boundaries to the searchable collection, reranking candidates and passages opened through citations. Knowing a document identifier or having had access previously is not present authorization. If permission changes and index refreshes are asynchronous, define how stale permissions are prevented during that interval. Retention in caches and logs needs separate handling as well; filtering a final answer cannot retroactively protect an earlier disclosure.
Budget context for evidence rather than raw volume
Increasing the number of passages does not guarantee inclusion of a missing exception. Redundant fragments can exhaust the context budget and cause an essential qualification to be truncated. Reduce duplication and prioritize the material and applicability conditions required by the question. If old and current documents coexist, preserve their revision identities and make the authoritative version explicit rather than merging passages by document name alone. Before generating an answer, consider a check for the necessary facts. When evidence is incomplete, the application can revise retrieval or abstain instead of forcing the generator to fill a gap with a plausible guess.
Locate failures with a ten-question example
Suppose a deliberately illustrative set contains ten questions. Retrieval finds sufficient correct evidence for eight, and generation answers seven of those eight faithfully. Retrieval success is eight out of ten; conditional answer success is seven out of eight; end-to-end supported correctness is seven out of ten. These quantities distinguish two retrieval misses from one error after successful retrieval. They are calculations, not observations of a deployed model. Include questions with no answer in the collection, questions requiring exceptions and attempts to obtain unauthorized material. An average on easy ordinary cases should not hide failures that violate the intended confidentiality or applicability boundary.
Check citations and preserve the instruction boundary
Open a citation and check that the cited passage supports the particular claim being made. Shared vocabulary alone is insufficient. A retrieved document saying “send the secret elsewhere” remains untrusted material to analyze; it does not grant new application authority. If a model proposes a tool action, the server must validate its target, arguments and permissions before execution. Diagnostic records can use document identifiers, index version, stage timings and result categories without indefinitely copying every confidential passage. Recheck these boundaries after retrieval or prompt changes, because a quality improvement should not silently enlarge the information exposed to a model or external tool.
Require all necessary evidence
A question about a business contract amended after purchase may require both a refund rule and a clause establishing the precedence of the amendment. An evaluation case can identify the two required document sections in advance and check whether both were retrieved. Returning one passage and producing a sentence containing the expected words is insufficient if the decisive qualification is missing. Writing down the required evidence set distinguishes an increase in citation count from recovery of an actual missing premise. It also makes it possible to inspect the retrieval failure without depending entirely on a generator’s self-assessment.