How to run paraphrase mining with Sentence Transformers

Near-duplicate text can hide inside support tickets, product titles, FAQ questions, and content snippets that use slightly different wording. Sentence Transformers paraphrase mining embeds each text item and returns the highest-scoring pairs, which helps a reviewer find likely duplicates without comparing every row by hand.

The paraphrase_mining() helper computes embeddings with a SentenceTransformer model and returns triplets in the form score, id1, id2. Each ID is a position in the input list, so retain the original texts or map those positions to stable record IDs before reviewing database or export data.

Similarity scores identify review candidates rather than guaranteed duplicates. Choose a threshold from labeled examples in the target domain, and reduce query_chunk_size or corpus_chunk_size if a larger corpus exceeds the available memory.

1. Install Sentence Transformers in the active Python environment.

   $ python -m pip install --upgrade sentence-transformers

   The first model load may download files from Hugging Face before printing results.
   Related: How to install Sentence Transformers with pip

2. Create paraphrase_mining.py with the imports and input corpus.

   paraphrase_mining.py

   from sentence_transformers import SentenceTransformer
   from sentence_transformers.util import paraphrase_mining
    
    
   sentences = [
       "Reset a customer password from the account portal.",
       "Reset a user password in the account portal.",
       "Deploy the billing service to production.",
       "Release the billing service to production.",
       "Archive the weekly database backup.",
       "Bake sourdough bread after the dough rises.",
   ]

   For a real review, the inline list can come from exported records. External record IDs remain necessary because the returned pair IDs refer to list positions.

3. Append the paraphrase-mining configuration to paraphrase_mining.py.

   model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2", device="cpu")
   pairs = paraphrase_mining(
       model,
       sentences,
       show_progress_bar=False,
       batch_size=16,
       top_k=3,
       max_pairs=8,
   )

   top_k limits candidates retained per sentence, while max_pairs caps the returned review queue.

4. Append the ranked-result validation to paraphrase_mining.py.

   for rank, (score, first_id, second_id) in enumerate(pairs[:4], start=1):
       print(f"{rank}. score={score:.4f}")
       print(f"   {first_id}: {sentences[first_id]}")
       print(f"   {second_id}: {sentences[second_id]}")
    
   expected_pair = frozenset({0, 1})
   top_pairs = {frozenset({first_id, second_id}) for _, first_id, second_id in pairs[:3]}
    
   if expected_pair not in top_pairs:
       raise SystemExit("FAIL: password-reset pair missing from top three results")
    
   print("PASS: password-reset pair found in top three results")

5. Run the completed script to confirm the known duplicate appears among the top three candidates.

   $ python paraphrase_mining.py
   1. score=0.9188
      0: Reset a customer password from the account portal.
      1: Reset a user password in the account portal.
   2. score=0.8725
      2: Deploy the billing service to production.
      3: Release the billing service to production.
   3. score=0.3629
      0: Reset a customer password from the account portal.
      2: Deploy the billing service to production.
   4. score=0.3608
      0: Reset a customer password from the account portal.
      3: Release the billing service to production.
   PASS: password-reset pair found in top three results

   A nonzero exit reports that the known duplicate fell outside the top three candidates. Lower-scoring pairs need manual review because similarity alone is not a merge decision.