How to run reranking evaluation with Sentence Transformers

Ranking metrics expose a retrieval failure that a general similarity score can hide: a relevant document appears below distractors for the same query. Sentence Transformers provides RerankingEvaluator to measure this ordering directly with labeled query, positive, and negative document sets.

The evaluator encodes each query and its candidates with one SentenceTransformer model, sorts the candidates by similarity, and calculates MAP, MRR@k, and NDCG@k. The configured at_k value controls the cutoff used by the rank-sensitive metrics.

A small support-domain sample keeps the first run quick and makes the expected ordering easy to inspect. A minimum NDCG@3 threshold turns a weak ranking into a nonzero process exit, while the CSV output preserves the metrics for later model comparisons.

1. Create reranking_eval.py with the evaluator imports and labeled candidate sets.

   reranking_eval.py

   from sentence_transformers import SentenceTransformer
   from sentence_transformers.sentence_transformer.evaluation import (
       RerankingEvaluator,
   )
    
    
   samples = [
       {
           "query": "How do I reset a forgotten password?",
           "positive": [
               "Reset a lost account password from the profile security page.",
           ],
           "negative": [
               "Generate quarterly revenue charts from a CSV export.",
               "Tune the database connection pool for a busy API server.",
               "Schedule a sales demo with the accounts team.",
           ],
       },
       {
           "query": "How can I rotate API keys?",
           "positive": [
               "Rotate API tokens before sharing a new integration.",
           ],
           "negative": [
               "Download a password reset email from account settings.",
               "Create a dashboard with monthly revenue charts.",
               "Archive old support tickets after closing a case.",
           ],
       },
   ]

   Each sample requires one query, at least one positive document, and at least one negative document. Held-out labels from the target corpus are appropriate for model comparisons; this small set is only a smoke test.

2. Append the CPU model definition immediately after the sample list.

   model = SentenceTransformer(
       "sentence-transformers/all-MiniLM-L6-v2",
       device="cpu",
   )

   The first model load may download files from Hugging Face. A production evaluation should load the same model revision and device configuration used by the retrieval application.

3. Append the named evaluator configuration below the model definition.

   evaluator = RerankingEvaluator(
       samples=samples,
       name="support-smoke",
       at_k=3,
       write_csv=True,
       show_progress_bar=False,
   )

   The name becomes part of each metric key, and at_k=3 produces MRR@3 and NDCG@3 values. CSV output is written only when an output_path is supplied during evaluation.

4. Append the evaluation call and threshold gate below the evaluator configuration.

   results = evaluator(model, output_path="reranking-results")
   threshold = 0.80
   primary_score = results[evaluator.primary_metric]
    
   print(f"primary metric: {evaluator.primary_metric}")
   print(f"primary score: {primary_score:.4f}")
   for key in sorted(results):
       print(f"{key}: {results[key]:.4f}")
    
   if primary_score < threshold:
       raise SystemExit(
           f"primary score below threshold {threshold:.2f}: "
           f"{primary_score:.4f}"
       )
    
   print(f"verification: PASS threshold {threshold:.2f} reached")

   primary_metric resolves to the named NDCG@3 result. The script exits before the PASS line when that score falls below the chosen acceptance threshold.

5. Run reranking_eval.py to require the labeled candidates to meet the NDCG@3 threshold.

   $ python reranking_eval.py
   primary metric: support-smoke_ndcg@3
   primary score: 1.0000
   support-smoke_map: 1.0000
   support-smoke_mrr@3: 1.0000
   support-smoke_ndcg@3: 1.0000
   verification: PASS threshold 0.80 reached