How to evaluate a sparse encoder with Sentence Transformers

Sparse encoders turn text into weighted vocabulary dimensions, but plausible token weights do not prove that a model ranks relevant documents above distractors. Retrieval evaluation pairs queries with labeled corpus IDs and measures the resulting order, exposing failures that a single embedding inspection can miss.

The SparseInformationRetrievalEvaluator accepts dictionaries for queries and corpus text plus a relevance map from each query ID to the document IDs that count as correct. Accuracy@1 checks whether a relevant document leads the ranking, while recall@3, MRR@3, and nDCG@3 measure coverage and ordering within the first three results.

The sample uses rasyosef/splade-tiny and caps each vector at 64 active dimensions so the evaluation remains small enough for a local smoke test. Perfect scores prove only that this labeled example is wired correctly; compare candidate models on a larger held-out dataset with representative distractors before choosing one.

1. Create sparse_encoder_evaluate.py with the sparse encoder and labeled retrieval data.

   sparse_encoder_evaluate.py

   from sentence_transformers import SparseEncoder
   from sentence_transformers.sparse_encoder.evaluation import (
       SparseInformationRetrievalEvaluator,
   )
    
   model = SparseEncoder("rasyosef/splade-tiny", device="cpu")
    
   queries = {
       "q1": "reset a forgotten password",
       "q2": "export invoices as CSV",
   }
    
   corpus = {
       "d1": "Reset a forgotten password from the account security page.",
       "d2": "Export invoices as a CSV file from the billing dashboard.",
       "d3": "Change the workspace color theme.",
       "d4": "Archive an inactive user account.",
   }
    
   relevant_docs = {
       "q1": {"d1"},
       "q2": {"d2"},
   }

   Every key in relevant_docs must identify an existing query, and every document ID in its set must exist in corpus.

2. Append the retrieval metrics and active-dimension cap to sparse_encoder_evaluate.py.

   evaluator = SparseInformationRetrievalEvaluator(
       queries=queries,
       corpus=corpus,
       relevant_docs=relevant_docs,
       name="support-search",
       accuracy_at_k=[1],
       precision_recall_at_k=[3],
       mrr_at_k=[3],
       ndcg_at_k=[3],
       map_at_k=[3],
       max_active_dims=64,
       show_progress_bar=False,
       write_csv=False,
   )

   max_active_dims=64 keeps the 64 highest-weight vocabulary dimensions in each sparse vector during evaluation. A representative validation set should show whether this cap changes ranking quality before deployment.

3. Append result calculation and metric reporting to sparse_encoder_evaluate.py.

   results = evaluator(model)
    
   for metric in (
       "support-search_dot_accuracy@1",
       "support-search_dot_recall@3",
       "support-search_dot_mrr@3",
       "support-search_dot_ndcg@3",
   ):
       print(f"{metric}: {results[metric]:.3f}")
    
   print(f"primary metric: {evaluator.primary_metric}")
   print(f"primary score: {results[evaluator.primary_metric]:.3f}")

   The evaluator prefixes result keys with the name value and sparse dot-product score function. Its primary metric resolves to nDCG@3 for these configured cutoffs.

4. Run the completed sparse encoder evaluation.

   $ python sparse_encoder_evaluate.py
   support-search_dot_accuracy@1: 1.000
   support-search_dot_recall@3: 1.000
   support-search_dot_mrr@3: 1.000
   support-search_dot_ndcg@3: 1.000
   primary metric: support-search_dot_ndcg@3
   primary score: 1.000

   The first model load may download files from Hugging Face before the metric lines appear. A lower score means that at least one labeled document missed a cutoff or ranked below its ideal position.