How to evaluate an embedding model on STS data with Sentence Transformers

Embedding models can produce plausible neighbors while still disagreeing with human judgments about how closely two sentences mean the same thing. Semantic textual similarity (STS) evaluation turns that disagreement into a repeatable correlation test for a Sentence Transformers model.

The EmbeddingSimilarityEvaluator class encodes two aligned sentence columns and compares model similarity with numeric reference scores. Cosine is the appropriate metric when downstream retrieval or ranking also uses cosine; the evaluator reports both Pearson association and Spearman rank agreement.

A small labeled set works as a smoke test for code integration, not as a benchmark. Release decisions need a held-out domain or STS validation split and an acceptance threshold chosen from model-selection results.

1. Create evaluate_sts.py with three aligned STS data lists.

   evaluate_sts.py

   from sentence_transformers import SentenceTransformer
   from sentence_transformers.sentence_transformer.evaluation import (
       EmbeddingSimilarityEvaluator,
   )
    
   sentences1 = [
       "A support agent reset the user's password.",
       "The database backup completed overnight.",
       "The release was rolled back after the deploy.",
       "The customer changed the billing address.",
       "The web server returned an SSL certificate warning.",
       "A new search index was built for documents.",
   ]
    
   sentences2 = [
       "The account password was reset by support.",
       "A nightly backup of the database finished successfully.",
       "The deployment was reverted after the release.",
       "A user updated payment and billing details.",
       "The office printer ran out of toner.",
       "The kitchen refrigerator needs cleaning.",
   ]
    
   scores = [0.95, 0.90, 0.86, 0.55, 0.05, 0.03]

   Each position describes one labeled pair across sentences1, sentences2, and scores. A held-out validation split must preserve that alignment.

2. Add a CPU model and cosine evaluator below the score list.

   model = SentenceTransformer(
       "sentence-transformers/all-MiniLM-L6-v2",
       device="cpu",
   )
    
   evaluator = EmbeddingSimilarityEvaluator(
       sentences1=sentences1,
       sentences2=sentences2,
       scores=scores,
       name="support-sts",
       main_similarity="cosine",
       show_progress_bar=False,
       write_csv=False,
   )

   The evaluator's similarity function should match the downstream application. Cosine is the default for many sentence-embedding retrieval systems.

3. Add the correlation report and acceptance threshold below the evaluator.

   results = evaluator(model)
   primary_metric = evaluator.primary_metric
   primary_score = results[primary_metric]
    
   print("Pairs evaluated:", len(sentences1))
   print("Primary metric:", primary_metric)
   print(f"Primary score: {primary_score:.3f}")
   print(f"Pearson cosine: {results['support-sts_pearson_cosine']:.3f}")
   print(f"Spearman cosine: {results['support-sts_spearman_cosine']:.3f}")
    
   if primary_score < 0.75:
       raise SystemExit("STS evaluator score is below the acceptance threshold")
    
   print("STS evaluation passed: primary score >= 0.750")

   The evaluator name becomes part of each result key. A score below the threshold exits with a failure message, so a project test or release job can stop instead of accepting the model.

4. Run the completed STS evaluator.

   $ python3 evaluate_sts.py
   Pairs evaluated: 6
   Primary metric: support-sts_spearman_cosine
   Primary score: 0.771
   Pearson cosine: 0.971
   Spearman cosine: 0.771
   STS evaluation passed: primary score >= 0.750

   The first model load can print Hugging Face download or weight-loading messages before the metric output. The final line appears only when the primary correlation meets the selected threshold.