How to rerank search results with a Sentence Transformers cross-encoder

Embedding retrieval can search a large corpus efficiently, but its similarity scores come from query and document vectors computed separately. A Sentence Transformers cross-encoder provides a slower second pass over a small candidate set so the final result order reflects each complete query-document pair.

The encode_query() and encode_document() methods keep the first-stage search roles explicit, while semantic_search() limits the expensive reranker to the selected top_k candidates. Retaining each original document ID alongside its text lets the application reconnect ranked candidates to database records, URLs, or other metadata.

The CrossEncoder.rank() method sorts its input documents by score and reports a corpus_id relative to that candidate list. The cross-encoder/ms-marco-MiniLM-L6-v2 model emits raw relevance logits, so their descending order determines the ranking without implying calibrated probabilities.

1. Create rerank_results.py with the query and searchable document collection.

   rerank_results.py

   from sentence_transformers import CrossEncoder, SentenceTransformer, util
    
   query = "How do I restore a Docker volume backup on another host?"
   documents = [
       "Create a tar archive from a Docker volume, copy it to the new host, and extract it into a replacement volume.",
       "Use docker compose pull and docker compose up -d to recreate application containers after an image update.",
       "List Docker images and tags before promoting a release to production.",
       "Upload a finished build directory to Amazon S3 with aws s3 sync for release backups.",
       "Inspect container logs with docker logs when a service exits during startup.",
   ]

2. Append the bi-encoder and cross-encoder model initialization to rerank_results.py.

   retriever = SentenceTransformer(
       "sentence-transformers/all-MiniLM-L6-v2",
       device="cpu",
   )
   reranker = CrossEncoder(
       "cross-encoder/ms-marco-MiniLM-L6-v2",
       device="cpu",
   )

   The first run downloads both models from the Hugging Face Hub. The portable example uses device=“cpu”; production workloads can select an available accelerator.

3. Append the embedding retrieval stage to rerank_results.py.

   document_embeddings = retriever.encode_document(
       documents,
       normalize_embeddings=True,
       show_progress_bar=False,
   )
   query_embedding = retriever.encode_query(
       query,
       normalize_embeddings=True,
       show_progress_bar=False,
   )
   hits = util.semantic_search(
       query_embedding,
       document_embeddings,
       top_k=4,
   )[0]
   candidates = [
       {
           "document_id": hit["corpus_id"],
           "retrieval_score": hit["score"],
           "text": documents[hit["corpus_id"]],
       }
       for hit in hits
   ]

   A larger top_k favors retrieval recall at the cost of more cross-encoder work because the second stage scores every selected query-document pair.

4. Append the cross-encoder ranking and result display to rerank_results.py.

   reranked = reranker.rank(
       query,
       [candidate["text"] for candidate in candidates],
       show_progress_bar=False,
   )
    
   print("Embedding candidates:")
   for candidate in candidates:
       print(
           f"document={candidate['document_id']} "
           f"cosine={candidate['retrieval_score']:.3f} | "
           f"{candidate['text']}"
       )
    
   print()
   print("Cross-encoder order:")
   for position, rank_hit in enumerate(reranked, start=1):
       candidate = candidates[rank_hit["corpus_id"]]
       print(
           f"{position}. document={candidate['document_id']} "
           f"score={rank_hit['score']:.2f} | "
           f"{candidate['text']}"
       )

   The reranker corpus_id points into candidates, while document_id preserves the corresponding position in the original collection.

5. Run the completed script to confirm the candidate documents are returned in descending cross-encoder order.

   $ python3 rerank_results.py
   Embedding candidates:
   document=0 cosine=0.740 | Create a tar archive from a Docker volume, copy it to the new host, and extract it into a replacement volume.
   document=1 cosine=0.466 | Use docker compose pull and docker compose up -d to recreate application containers after an image update.
   document=4 cosine=0.364 | Inspect container logs with docker logs when a service exits during startup.
   document=2 cosine=0.272 | List Docker images and tags before promoting a release to production.
   
   Cross-encoder order:
   1. document=0 score=3.19 | Create a tar archive from a Docker volume, copy it to the new host, and extract it into a replacement volume.
   2. document=1 score=-6.19 | Use docker compose pull and docker compose up -d to recreate application containers after an image update.
   3. document=4 score=-7.50 | Inspect container logs with docker logs when a service exits during startup.
   4. document=2 score=-10.33 | List Docker images and tags before promoting a release to production.