How to add reranking to a LlamaIndex chatbot

A retrieval-backed chatbot can find relevant passages yet send them to the language model in an unhelpful order. LlamaIndex node postprocessors provide a second ranking stage between retrieval and response synthesis, allowing the chatbot to keep a broader candidate set while limiting its final context to the strongest matches.

The context chat mode accepts node postprocessors through node_postprocessors when as_chat_engine() creates the engine. Retrieval first supplies three candidates, and the reranker reduces that set to the two ticket-specific passages before the custom language model receives the prompt.

API credentials and model downloads are unnecessary for the executable example because MockEmbedding, a deterministic BaseNodePostprocessor implementation, and a small custom language model run entirely in process. A production chatbot can replace TicketReranker with a cross-encoder or hosted reranker while retaining the same node_postprocessors hook.

1. Install LlamaIndex core in the chatbot's active Python environment.

   $ python3 -m pip install llama-index-core

2. Create rerank_chatbot.py with the imports and ticket reranker.

   rerank_chatbot.py

   from llama_index.core import Document, Settings, VectorStoreIndex
   from llama_index.core.embeddings import MockEmbedding
   from llama_index.core.llms import CompletionResponse, CustomLLM, LLMMetadata
   from llama_index.core.llms.callbacks import llm_completion_callback
   from llama_index.core.postprocessor.types import BaseNodePostprocessor
    
    
   class TicketReranker(BaseNodePostprocessor):
       top_n: int = 2
    
       def _postprocess_nodes(self, nodes, query_bundle=None):
           query = query_bundle.query_str.lower() if query_bundle else ""
    
           for item in nodes:
               text = item.node.get_content().lower()
               score = 0.0
               score += 2.0 if "ticket 7421" in text else 0.0
               score += 2.0 if "who owns" in query and "belongs to" in text else 0.0
               score += 1.0 if "escalated" in query and "escalated" in text else 0.0
               item.score = score
    
           return sorted(nodes, key=lambda item: item.score or 0.0, reverse=True)[
               : self.top_n
           ]

   BaseNodePostprocessor supplies postprocess_nodes() and passes the active query to _postprocess_nodes(). The implementation assigns higher scores to the owner and escalation passages, then returns only top_n nodes.

3. Add the test language model below TicketReranker.

   rerank_chatbot.py

   class SupportBotLLM(CustomLLM):
       @property
       def metadata(self) -> LLMMetadata:
           return LLMMetadata(
               context_window=2048,
               num_output=64,
               model_name="support-bot-llm",
           )
    
       def answer_from_prompt(self, prompt: str) -> str:
           prompt_text = prompt.lower()
           has_owner = "ticket 7421 belongs to maya" in prompt_text
           has_queue = "docs-review queue" in prompt_text
           if has_owner and has_queue:
               return "owner=Maya; escalation=docs-review"
           return "missing ticket owner or escalation queue"
    
       @llm_completion_callback()
       def complete(self, prompt: str, formatted: bool = False, **kwargs):
           return CompletionResponse(text=self.answer_from_prompt(prompt))
    
       @llm_completion_callback()
       def stream_complete(self, prompt: str, formatted: bool = False, **kwargs):
           text = self.answer_from_prompt(prompt)
           for token in text.split():
               yield CompletionResponse(text=text, delta=f"{token} ")

   The test model returns the successful answer only when both reranked facts appear in its prompt, so the final response can fail when either passage is missing.

4. Add the model settings and support documents below SupportBotLLM.

   rerank_chatbot.py

   Settings.llm = SupportBotLLM()
   Settings.embed_model = MockEmbedding(embed_dim=8)
    
   documents = [
       Document(
           text="General support chatbot style: keep responses short.",
           metadata={"source": "chatbot-style"},
       ),
       Document(
           text="Ticket 7421 belongs to Maya in billing operations.",
           metadata={"source": "ticket-owner"},
       ),
       Document(
           text="Answers for ticket 7421 should be escalated to the docs-review queue.",
           metadata={"source": "ticket-escalation"},
       ),
   ]

5. Add the index and reranking stage below documents.

   rerank_chatbot.py

   index = VectorStoreIndex.from_documents(documents)
   question = "For ticket 7421, who owns it and where should answers be escalated?"
   reranker = TicketReranker(top_n=2)
    
   retrieved_nodes = index.as_retriever(similarity_top_k=3).retrieve(question)
   reranked_nodes = reranker.postprocess_nodes(retrieved_nodes, query_str=question)

   A similarity_top_k value larger than top_n gives the reranker more candidates than it returns.

6. Append the chat engine and output checks below reranked_nodes.

   rerank_chatbot.py

   chat_engine = index.as_chat_engine(
       chat_mode="context",
       similarity_top_k=3,
       node_postprocessors=[reranker],
   )
   response = chat_engine.chat(question)
    
   print("reranked_sources=" + ",".join(item.node.metadata["source"] for item in reranked_nodes))
   print(f"reranked_count={len(reranked_nodes)}")
   print(f"answer={response}")

   The same reranker is passed to the chat engine, so its response uses the postprocessed node set rather than the original three retrieval candidates.

7. Run the chatbot to confirm that reranking retains both ticket passages and produces the complete answer.

   $ python3 rerank_chatbot.py
   reranked_sources=ticket-owner,ticket-escalation
   reranked_count=2
   answer=owner=Maya; escalation=docs-review