How to score text pairs with a Sentence Transformers cross-encoder

A CrossEncoder evaluates both sides of a text pair together, which suits candidate sets that need a direct relevance judgment rather than reusable embeddings. Sentence Transformers exposes that joint inference path through CrossEncoder.predict().

The cross-encoder/ms-marco-MiniLM-L6-v2 model is trained for query-passage ranking and returns raw logits by default. Applying a sigmoid maps those logits to scores between 0 and 1 without changing their order, but the values remain model-specific rather than universal relevance thresholds.

One query with two relevant passages and one unrelated passage makes the expected ordering visible. The script exits with an error if either relevant passage falls below the unrelated passage, so a successful run checks the model output instead of merely printing fixed values.

1. Create score_pairs.py with one query and three candidate passages.

   score_pairs.py

   import torch
   from sentence_transformers import CrossEncoder
    
   query = "How can I reset a forgotten password?"
   passages = [
       "Open account settings and request a password reset link.",
       "Contact the support desk to verify your identity and reset the password.",
       "Rotate the TLS certificate before the next web server deployment.",
   ]
   pairs = [(query, passage) for passage in passages]

2. Append the cross-encoder model and pair-scoring call after the pairs assignment.

   model = CrossEncoder(
       "cross-encoder/ms-marco-MiniLM-L6-v2",
       device="cpu",
       activation_fn=torch.nn.Sigmoid(),
   )
   scores = model.predict(pairs, show_progress_bar=False)

   The sigmoid keeps the ranking unchanged while making the displayed scores easier to compare. The selected MS MARCO reranker expects query-passage input; models trained for other pair relationships can use a different score scale.

3. Append the ordered score display and fail-capable relevance check after the scores assignment.

   for score, passage in sorted(
       zip(scores, passages),
       key=lambda item: float(item[0]),
       reverse=True,
   ):
       print(f"{float(score):.4f}  {passage}")
    
   if min(float(scores[0]), float(scores[1])) <= float(scores[2]):
       raise SystemExit("A relevant passage did not score above the unrelated passage.")
    
   print("check: both relevant passages scored above the unrelated passage")

   A numeric threshold needs labeled pairs from the intended application; the relative-order assertion avoids treating one demonstration score as a universal cutoff.

4. Run score_pairs.py to score all three query-passage pairs.

   $ python3 score_pairs.py
   0.9240  Contact the support desk to verify your identity and reset the password.
   0.8037  Open account settings and request a password reset link.
   0.0000  Rotate the TLS certificate before the next web server deployment.
   check: both relevant passages scored above the unrelated passage