How to score text pairs with a Sentence Transformers cross-encoder

A Sentence Transformers cross-encoder reads both texts in a pair at the same time, so it can judge whether a query and passage match without saving embeddings for later search. Pair scoring is useful when an application already has a small candidate set, a moderation queue, or labeled examples that need direct relevance scores.

CrossEncoder.predict() accepts a list of two-text pairs and returns one score per pair. The cross-encoder/ms-marco-MiniLM-L6-v2 model is trained for query-passage relevance, so higher scores mean the passage is a better answer for that query rather than a reusable vector distance.

A small local check keeps one intentionally unrelated pair beside two relevant pairs so the score order can be checked by eye. The first model load may download weights or print progress messages before the score table, and application thresholds should be chosen from validation data instead of copied from one demonstration run.

1. Create a Python script that scores query-passage pairs with CrossEncoder.predict().

   score_pairs.py

   from sentence_transformers import CrossEncoder
    
   pairs = [
       (
           "How do I reset a user password?",
           "Use the account settings page to send a password reset link.",
       ),
       (
           "How do I reset a user password?",
           "Rotate the signing key before deploying a new release.",
       ),
       (
           "Which document explains database backups?",
           "Create a snapshot, copy it to archive storage, and record the restore point.",
       ),
   ]
    
   model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2", device="cpu")
   scores = model.predict(pairs, show_progress_bar=False)
    
   ranked_pairs = sorted(
       zip(scores, pairs),
       key=lambda item: float(item[0]),
       reverse=True,
   )
    
   for rank, (score, (query, passage)) in enumerate(ranked_pairs, start=1):
       print(f"{rank}. score={float(score):.2f}")
       print(f"   query: {query}")
       print(f"   passage: {passage}")
    
   if float(scores[0]) <= float(scores[1]):
       raise SystemExit("password reset pair did not score above unrelated release text")
    
   if float(scores[2]) <= float(scores[1]):
       raise SystemExit("backup pair did not score above unrelated release text")
    
   print("check: relevant pairs scored above the unrelated pair")

   Use a cross-encoder model that matches the pair type. MS MARCO cross-encoders are query-passage rerankers; semantic textual similarity cross-encoders can use a different score scale.

2. Run the pair scoring script.

   $ python3 score_pairs.py
   1. score=4.34
      query: How do I reset a user password?
      passage: Use the account settings page to send a password reset link.
   2. score=-8.78
      query: Which document explains database backups?
      passage: Create a snapshot, copy it to archive storage, and record the restore point.
   3. score=-10.36
      query: How do I reset a user password?
      passage: Rotate the signing key before deploying a new release.
   check: relevant pairs scored above the unrelated pair

   The first run can print model download or weight-loading lines before the score table. Keep the application check tied to relative ordering unless a labeled validation set establishes a numeric threshold.

3. Check that each relevant pair scores above the unrelated pair.

   The password-reset pair ranks first for the password query, and the database-backup pair still scores above the unrelated release text. A failed SystemExit message means the chosen model or pair formatting does not match the task.

4. Remove the temporary scoring script when the check is finished.

   $ rm score_pairs.py