How to run retrieval evaluation in Haystack

Retriever changes alter which evidence reaches a search result or RAG prompt, and aggregate evaluation makes those changes measurable across a labeled query set. Haystack can calculate recall and mean reciprocal rank from captured retrieval results before a new retriever configuration is adopted.

Haystack's DocumentRecallEvaluator reports whether each query retrieved a labeled document in its default single-hit mode. DocumentMRREvaluator also considers the first relevant document's position, so a match at rank two receives a lower reciprocal-rank score than a match at rank one.

Ground-truth and retrieved document lists must use the same query order. Comparing stable Document.id values avoids treating harmless content edits as different labels, while retaining the retriever's original result order preserves the ranking signal measured by MRR.

1. Create retrieval_evaluation.py with the evaluator imports and one labeled result set per query.

   retrieval_evaluation.py

   from haystack import Document, Pipeline
   from haystack.components.evaluators import DocumentMRREvaluator, DocumentRecallEvaluator
    
    
   def document(document_id: str) -> Document:
       return Document(id=document_id)
    
    
   ground_truth_documents = [
       [document("refund-policy")],
       [document("support-hours")],
       [document("shipping-region")],
   ]
    
   retrieved_documents = [
       [document("refund-policy"), document("support-hours")],
       [document("shipping-region"), document("support-hours")],
       [document("refund-policy"), document("support-hours")],
   ]

   The retrieved_documents value represents saved output from the retriever under evaluation. Its document order remains unchanged, and both outer lists use the same query order.

2. Append the evaluation pipeline and ID-based recall component after retrieved_documents.

   retrieval_evaluation.py

   evaluation_pipeline = Pipeline()
   evaluation_pipeline.add_component(
       "recall", DocumentRecallEvaluator(document_comparison_field="id")
   )

   The default single_hit mode scores a query as 1.0 when any labeled document is retrieved. The multi_hit mode instead reflects how many labeled documents were retrieved for each query.

3. Append the ID-based MRR component after the recall component.

   retrieval_evaluation.py

   evaluation_pipeline.add_component(
       "mrr", DocumentMRREvaluator(document_comparison_field="id")
   )

4. Append the shared evaluator inputs and pipeline run after the MRR component.

   retrieval_evaluation.py

   evaluator_inputs = {
       "ground_truth_documents": ground_truth_documents,
       "retrieved_documents": retrieved_documents,
   }
   results = evaluation_pipeline.run(
       {"recall": evaluator_inputs, "mrr": evaluator_inputs}
   )

5. Append the score reporting after the pipeline run.

   retrieval_evaluation.py

   print("recall by query:", results["recall"]["individual_scores"])
   print("MRR by query:", results["mrr"]["individual_scores"])
   print(f"mean recall: {results['recall']['score']:.2f}")
   print(f"mean MRR: {results['mrr']['score']:.2f}")

6. Run the completed retrieval evaluation to verify the labeled query scores in the haystack-ai environment.

   $ python retrieval_evaluation.py
   recall by query: [1.0, 1.0, 0.0]
   MRR by query: [1.0, 0.5, 0.0]
   mean recall: 0.67
   mean MRR: 0.50

   The second query receives full recall because its labeled document was retrieved, but its MRR is 0.5 because that document appears at rank two. Both scores are 0.0 for the third query because its label is absent.