How to build hybrid search with Sentence Transformers

Search queries often mix descriptive language with identifiers that only exact-term matching can recognize. A hybrid retriever keeps both signals by combining dense semantic ranking from Sentence Transformers with lexical BM25 ranking before presenting one result list.

The dense branch uses encode_document() and encode_query() with normalized embeddings, while BM25Okapi applies the same lowercase tokenization to the documents and query. Each record keeps its application-owned ID beside the searchable text so the fused positions can be mapped back to source data.

Reciprocal rank fusion adds contributions from the dense and lexical positions instead of mixing their raw scores, which use different scales. The final check requires both the semantic description in doc-001 and the exact policy identifier in doc-002 to remain in the first three fused results.

1. Install Sentence Transformers and rank-bm25 in the active Python environment.

   $ python -m pip install --upgrade sentence-transformers rank-bm25

   The first model run may download files from Hugging Face before printing the ranking.

2. Create hybrid_search.py with the imports and retrieval inputs.

   hybrid_search.py

   import re
   from collections import Counter
    
   import numpy as np
   from rank_bm25 import BM25Okapi
   from sentence_transformers import SentenceTransformer, util
    
    
   documents = [
       {
           "id": "doc-001",
           "text": "Customers can request money back when their card is charged twice.",
       },
       {
           "id": "doc-002",
           "text": "Policy ACME-482 covers duplicate invoice refunds.",
       },
       {
           "id": "doc-003",
           "text": "Export monthly invoices from the billing dashboard.",
       },
       {
           "id": "doc-004",
           "text": "Reset an expired password from the account security page.",
       },
       {
           "id": "doc-005",
           "text": "Configure alerts for failed card payments.",
       },
   ]
   query = "ACME-482 card charged two times"

3. Add shared tokenization and BM25 scoring below the query assignment.

   def tokenize(text):
       return re.findall(r"[a-z0-9]+", text.lower())
    
    
   corpus = [document["text"] for document in documents]
   bm25 = BM25Okapi([tokenize(text) for text in corpus])
   lexical_scores = bm25.get_scores(tokenize(query))

   rank-bm25 expects preprocessed tokens, so the documents and query must pass through the same function.

4. Add the dense retrieval branch below the lexical scores.

   model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
   document_embeddings = model.encode_document(
       corpus,
       normalize_embeddings=True,
       convert_to_tensor=True,
       show_progress_bar=False,
   )
   query_embedding = model.encode_query(
       query,
       normalize_embeddings=True,
       convert_to_tensor=True,
       show_progress_bar=False,
   )
   dense_scores = util.dot_score(query_embedding, document_embeddings)[0].cpu().numpy()

   encode_query() and encode_document() select distinct prompts or routes when a retrieval model defines them.

5. Add the ranking and reciprocal-rank-fusion helpers below the dense scores.

   def top_hits(scores, limit=3):
       order = np.argsort(scores)[::-1][:limit]
       return [int(index) for index in order]
    
    
   def reciprocal_rank_fusion(dense_hits, lexical_hits, k=60):
       fused_scores = Counter()
       dense_ranks = {}
       lexical_ranks = {}
    
       for rank, index in enumerate(dense_hits, start=1):
           fused_scores[index] += 1 / (k + rank)
           dense_ranks[index] = rank
    
       for rank, index in enumerate(lexical_hits, start=1):
           fused_scores[index] += 1 / (k + rank)
           lexical_ranks[index] = rank
    
       return [
           (index, score, dense_ranks.get(index, "-"), lexical_ranks.get(index, "-"))
           for index, score in fused_scores.most_common()
       ]

   The k constant reduces the effect of a one-position difference while still rewarding records returned by both branches.

6. Append fused retrieval and fail-capable output below the fusion helper.

   dense_hits = top_hits(dense_scores)
   lexical_hits = top_hits(lexical_scores)
   fused_hits = reciprocal_rank_fusion(dense_hits, lexical_hits)[:3]
    
   print(f"Query: {query}")
   print("Fused ranking:")
   for rank, (index, score, dense_rank, lexical_rank) in enumerate(fused_hits, start=1):
       document = documents[index]
       print(
           f"{rank}. {document['id']} rrf={score:.4f} "
           f"dense_rank={dense_rank} lexical_rank={lexical_rank}"
       )
    
   returned_ids = {documents[index]["id"] for index, *_ in fused_hits}
   required_ids = {"doc-001", "doc-002"}
   if not required_ids.issubset(returned_ids):
       raise SystemExit(f"Hybrid search check: failed; returned {sorted(returned_ids)}")
    
   print("Hybrid search check: passed")

7. Run the completed hybrid search program.

   $ python hybrid_search.py
   Query: ACME-482 card charged two times
   Fused ranking:
   1. doc-001 rrf=0.0325 dense_rank=1 lexical_rank=2
   2. doc-002 rrf=0.0325 dense_rank=2 lexical_rank=1
   3. doc-005 rrf=0.0317 dense_rank=3 lexical_rank=3
   Hybrid search check: passed