How to run a RAG evaluation in LlamaIndex

Retrieval quality can regress even when a RAG application still returns fluent answers. LlamaIndex retrieval evaluation turns labeled queries and expected node IDs into repeatable measurements, exposing whether the intended context appears and how highly it ranks.

The RetrieverEvaluator runs each query through the configured retriever. hit_rate records whether an expected node appears in the retrieved set, while mrr rewards an expected node for appearing closer to rank one.

The deterministic KeywordEmbedding below keeps the exercise local and makes both a rank-one match and a rank-two match reproducible. Use the application embedding model and representative labeled queries for production evaluation; retrieval metrics do not assess whether generated answers are faithful or relevant.

1. Create evaluate_rag.py with the imports and deterministic embedding fixture.

   evaluate_rag.py

   from statistics import mean
    
   from llama_index.core import VectorStoreIndex
   from llama_index.core.embeddings import BaseEmbedding
   from llama_index.core.evaluation import RetrieverEvaluator
   from llama_index.core.schema import TextNode
    
    
   class KeywordEmbedding(BaseEmbedding):
       def _vector_for_text(self, text: str) -> list[float]:
           text = text.lower()
           if "billing" in text and "release" in text:
               return [0.8, 0.6, 0.0]
           if "billing" in text or "7421" in text:
               return [1.0, 0.0, 0.0]
           if "release" in text:
               return [0.0, 1.0, 0.0]
           return [0.0, 0.0, 1.0]
    
       def _get_text_embedding(self, text: str) -> list[float]:
           return self._vector_for_text(text)
    
       def _get_query_embedding(self, query: str) -> list[float]:
           return self._vector_for_text(query)
    
       async def _aget_query_embedding(self, query: str) -> list[float]:
           return self._get_query_embedding(query)

   The active project environment needs llama-index-core for these imports. The fixture deliberately maps a mixed billing-and-release query closer to the billing node so the evaluation exposes a rank-two expected result.

2. Append the labeled nodes and retriever below the KeywordEmbedding class.

   nodes = [
       TextNode(
           id_="billing-escalation",
           text="Escalate stale billing answers for ticket 7421 to docs review.",
       ),
       TextNode(
           id_="release-freeze",
           text="Release freeze exceptions require approval from the release desk.",
       ),
       TextNode(
           id_="password-reset",
           text="Password reset requests require identity verification.",
       ),
   ]
    
   index = VectorStoreIndex(nodes, embed_model=KeywordEmbedding())
   retriever = index.as_retriever(similarity_top_k=2)

   Stable id_ values connect the expected labels to the nodes returned by the retriever. Durable production document or chunk IDs prevent label drift when display text changes.

3. Append the evaluator and labeled queries below the retriever.

   evaluator = RetrieverEvaluator.from_metric_names(
       ["hit_rate", "mrr"],
       retriever=retriever,
   )
    
   labeled_queries = [
       ("Where should billing answers for ticket 7421 be escalated?", ["billing-escalation"]),
       ("Who approves billing changes during the release freeze?", ["release-freeze"]),
   ]

   Each label pairs one query with the node IDs that should satisfy it. The second query expects release-freeze even though the fixture ranks billing-escalation first.

4. Append the evaluation runner below labeled_queries.

   results = []
   for query, expected_ids in labeled_queries:
       result = evaluator.evaluate(query=query, expected_ids=expected_ids)
       results.append(result)
       print(f"query: {query}")
       print(f"expected_ids: {expected_ids}")
       print(f"retrieved_ids: {result.retrieved_ids}")
       print(f"hit_rate: {result.metric_vals_dict['hit_rate']}")
       print(f"mrr: {result.metric_vals_dict['mrr']}")
       print()
    
   mean_hit_rate = mean(result.metric_vals_dict["hit_rate"] for result in results)
   mean_mrr = mean(result.metric_vals_dict["mrr"] for result in results)
   passed = mean_hit_rate == 1.0 and mean_mrr >= 0.75
    
   print(f"mean_hit_rate: {mean_hit_rate}")
   print(f"mean_mrr: {mean_mrr}")
   print(f"status: {'PASS' if passed else 'FAIL'}")

   The sample threshold requires every expected node to appear in the top two results and permits one expected node to rank second. Production thresholds depend on the accepted application baseline.

5. Compare evaluate_rag.py with the completed evaluation program.

   evaluate_rag.py

   from statistics import mean
    
   from llama_index.core import VectorStoreIndex
   from llama_index.core.embeddings import BaseEmbedding
   from llama_index.core.evaluation import RetrieverEvaluator
   from llama_index.core.schema import TextNode
    
    
   class KeywordEmbedding(BaseEmbedding):
       def _vector_for_text(self, text: str) -> list[float]:
           text = text.lower()
           if "billing" in text and "release" in text:
               return [0.8, 0.6, 0.0]
           if "billing" in text or "7421" in text:
               return [1.0, 0.0, 0.0]
           if "release" in text:
               return [0.0, 1.0, 0.0]
           return [0.0, 0.0, 1.0]
    
       def _get_text_embedding(self, text: str) -> list[float]:
           return self._vector_for_text(text)
    
       def _get_query_embedding(self, query: str) -> list[float]:
           return self._vector_for_text(query)
    
       async def _aget_query_embedding(self, query: str) -> list[float]:
           return self._get_query_embedding(query)
    
    
   nodes = [
       TextNode(
           id_="billing-escalation",
           text="Escalate stale billing answers for ticket 7421 to docs review.",
       ),
       TextNode(
           id_="release-freeze",
           text="Release freeze exceptions require approval from the release desk.",
       ),
       TextNode(
           id_="password-reset",
           text="Password reset requests require identity verification.",
       ),
   ]
    
   index = VectorStoreIndex(nodes, embed_model=KeywordEmbedding())
   retriever = index.as_retriever(similarity_top_k=2)
    
   evaluator = RetrieverEvaluator.from_metric_names(
       ["hit_rate", "mrr"],
       retriever=retriever,
   )
    
   labeled_queries = [
       ("Where should billing answers for ticket 7421 be escalated?", ["billing-escalation"]),
       ("Who approves billing changes during the release freeze?", ["release-freeze"]),
   ]
    
   results = []
   for query, expected_ids in labeled_queries:
       result = evaluator.evaluate(query=query, expected_ids=expected_ids)
       results.append(result)
       print(f"query: {query}")
       print(f"expected_ids: {expected_ids}")
       print(f"retrieved_ids: {result.retrieved_ids}")
       print(f"hit_rate: {result.metric_vals_dict['hit_rate']}")
       print(f"mrr: {result.metric_vals_dict['mrr']}")
       print()
    
   mean_hit_rate = mean(result.metric_vals_dict["hit_rate"] for result in results)
   mean_mrr = mean(result.metric_vals_dict["mrr"] for result in results)
   passed = mean_hit_rate == 1.0 and mean_mrr >= 0.75
    
   print(f"mean_hit_rate: {mean_hit_rate}")
   print(f"mean_mrr: {mean_mrr}")
   print(f"status: {'PASS' if passed else 'FAIL'}")

6. Run the completed evaluation program in the project environment.

   $ python3 evaluate_rag.py
   query: Where should billing answers for ticket 7421 be escalated?
   expected_ids: ['billing-escalation']
   retrieved_ids: ['billing-escalation', 'release-freeze']
   hit_rate: 1.0
   mrr: 1.0
   
   query: Who approves billing changes during the release freeze?
   expected_ids: ['release-freeze']
   retrieved_ids: ['billing-escalation', 'release-freeze']
   hit_rate: 1.0
   mrr: 0.5
   
   mean_hit_rate: 1.0
   mean_mrr: 0.75
   status: PASS

   The second query still records a hit because release-freeze is present in the top two results, while mrr falls to 0.5 because that node is ranked second. The aggregate passes the sample threshold without hiding the ranking weakness.