Experiments and implementations · Intermediate

Measuring embedding-retrieval recall

Rank fixed small vectors against explicit relevance judgments and calculate per-query and aggregated recall independently of vector similarity.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. Define a controlled vector model
  2. Specify metric and tie breaking
  3. Compute metrics from IDs
  4. Check aggregation rules
  5. Separate representation and index changes
  6. Report evidence coverage
  7. Ask Logeia about this articleCheck your understanding

Define a controlled vector model

Assign two-dimensional vectors to documents A through D and search with two queries. This is a controlled numeric model for checking an evaluation procedure, not a benchmark of a trained embedding model. Declare relevance sets{A,C} and{C} separately from vector geometry. Deliberately allowing a nearby but irrelevant document demonstrates that scores and judgments provide different information. In a real evaluation, judgments must come from the question and supporting corpus rather than being inferred from the embedding scores themselves. Keeping that distinction visible prevents a circular evaluation that labels every retrieved neighbor correct by definition.

Specify metric and tie breaking

Rank by descending cosine similarity, breaking ties by ascending document ID. No zero vectors occur, and exhaustive comparison removes approximate-index search error from the experiment. Retrieve the top two, convert IDs to a set, and intersect with declared relevance. Save the code as recall.py and run it with Python 3’s standard math module. No model or dataset download is required. It changes no external state, so cleanup only concerns the script. Explicit metric, candidate set, tie rule, and k make the resulting rankings reproducible and keep evaluation arithmetic separate from index implementation.

Compute metrics from IDs

The first query ranks A and B at the top, retrieving only A from its two relevant documents. Precision and recall are both 0.5. The second retrieves C and B, giving recall one and precision 0.5. B remains irrelevant under the declared judgments despite geometric proximity. Assertions check recalls[0.5,1]. The metrics come from intersections of ID sets, not from interpreting similarity scores as correctness frequencies. This is an important separation: a high score affects ordering, while the relevance judgments decide whether that ordering supplies useful evidence for the question.

python
import math
vectors = {'A': (0.9,0.1), 'B': (0.8,0.2), 'C': (0.0,1.0), 'D': (-1.0,0.0)}
queries = [((1.0,0.0), {'A','C'}), ((0.0,1.0), {'C'})]
def cosine(a,b):
    return sum(x*y for x,y in zip(a,b))/(math.sqrt(sum(x*x for x in a))*math.sqrt(sum(y*y for y in b)))
recalls, hits_total, relevant_total = [], 0, 0
for query, relevant in queries:
    ranking = sorted(vectors, key=lambda key: (-cosine(query,vectors[key]),key))
    retrieved = set(ranking[:2])
    hits = len(retrieved & relevant)
    precision, recall = hits/len(retrieved), hits/len(relevant)
    print(ranking, precision, recall)
    recalls.append(recall)
    hits_total += hits
    relevant_total += len(relevant)
print('macro/micro recall', sum(recalls)/len(recalls), hits_total/relevant_total)
assert recalls == [0.5,1.0]
assert sum(recalls)/len(recalls) == 0.75

Check aggregation rules

Equal averaging of the two recalls gives macro recall 0.75. Pooling two relevant hits over three relevant documents gives micro recall two thirds. The difference is weighting, not rounding error. Questions with more relevant documents have greater influence under pooled counting. Retain per-query values and denominators so alternative aggregates can be reproduced. Adding questions with empty relevance sets requires a declared policy rather than a convenient silent choice after seeing results. The experiment’s simplicity makes these aggregation decisions visible before they are hidden inside a larger retrieval evaluation framework.

Separate representation and index changes

Because every candidate is scored, the first query’s result is not caused by an approximate index missing C. It follows from the chosen representation and separate relevance judgments. If an approximate index is later introduced, compare its results with the same exhaustive-ranking reference to isolate search approximation. Changing query or document models also raises representation-space compatibility issues. Vary k separately from changing the vectors so that a recall difference has an identifiable cause. A retrieval failure can originate in corpus coverage, representation, indexing, or evaluation definitions; this controlled example removes several of those factors.

Report evidence coverage

For k equal to one, two, and three, report rankings, retrieved IDs, precision, and recall for both queries. Verify that recall cannot fall when extending the prefix of the same fixed ranking. Completion means explaining denominators and aggregation without defining relevance solely by geometric proximity. Replacing fixed vectors with a trained model requires recording model version, corpus, queries, judgments, metric, and k. If a generator is added, retain retrieval results so failures despite sufficient evidence remain distinguishable from missing-evidence failures. That makes subsequent improvements assessable at the stage they actually change.

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Prepare for AI systems · STEP 16

Choose a learning path