How to audit knowledge base gaps in Haystack

Knowledge-base search can conceal missing documentation because a retriever still returns the nearest document when the corpus does not answer a question. A score-based audit in Haystack exposes those weak matches before a support bot or RAG pipeline treats them as usable context.

An InMemoryDocumentStore and InMemoryBM25Retriever provide a local audit surface without an external model or database. The audit labels a question FOUND only when its highest-scoring document reaches a threshold calibrated with questions the same knowledge base should and should not cover.

BM25 scores vary with corpus size, wording, and document length. A representative export and human review of each MISSING question keep unrelated top matches from becoming automatic writing assignments.

1. Create audit_kb_gaps.py with the Haystack imports and representative knowledge-base documents.

   audit_kb_gaps.py

   from haystack import Document
   from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
   from haystack.document_stores.in_memory import InMemoryDocumentStore
    
    
   documents = [
       Document(
           content="Users reset a forgotten password from Account settings with the password reset email.",
           meta={"title": "Password reset"},
       ),
       Document(
           content="Admins configure SAML single sign-on from Security settings with the identity provider metadata URL.",
           meta={"title": "SAML SSO setup"},
       ),
       Document(
           content="Billing owners change invoice recipients from Billing settings before the next invoice is issued.",
           meta={"title": "Invoice recipients"},
       ),
   ]

2. Add covered and uncovered audit questions with an initial score threshold below documents.

   questions = [
       "How do users reset a forgotten password?",
       "How do admins configure SAML single sign-on?",
       "Can I export a SOC 2 audit report?",
       "How can billing owners change invoice recipients?",
   ]
    
   minimum_score = 4.0

   A minimum_score calibrated with known covered and uncovered questions reflects the score distribution of the knowledge base being audited.

3. Add the in-memory document store and top-match BM25 retriever below minimum_score.

   document_store = InMemoryDocumentStore()
   document_store.write_documents(documents)
   retriever = InMemoryBM25Retriever(document_store=document_store, top_k=1)

4. Define audit_question() below the retriever to retain each question, score, status, and top document.

   def audit_question(question):
       matches = retriever.run(query=question)["documents"]
       top_document = matches[0] if matches else None
       score = top_document.score if top_document else 0.0
    
       return {
           "question": question,
           "status": "FOUND" if top_document and score >= minimum_score else "MISSING",
           "score": score,
           "title": top_document.meta["title"] if top_document else "none",
       }

5. Finish audit_kb_gaps.py with the per-question report and missing-question summary.

   results = [audit_question(question) for question in questions]
    
   for result in results:
       print(
           f"{result['status']} | score={result['score']:.2f} | "
           f"question={result['question']}"
       )
       print(f"top_document={result['title']}")
       print()
    
   missing_questions = [
       result["question"] for result in results if result["status"] == "MISSING"
   ]
    
   print(f"summary_found={len(results) - len(missing_questions)}")
   print(f"summary_missing={len(missing_questions)}")
   print("missing_questions=" + "; ".join(missing_questions))

6. Run the completed knowledge-base audit with the active Haystack Python environment.

   $ python3 audit_kb_gaps.py
   FOUND | score=6.86 | question=How do users reset a forgotten password?
   top_document=Password reset
   
   FOUND | score=7.23 | question=How do admins configure SAML single sign-on?
   top_document=SAML SSO setup
   
   MISSING | score=1.25 | question=Can I export a SOC 2 audit report?
   top_document=Password reset
   
   FOUND | score=6.74 | question=How can billing owners change invoice recipients?
   top_document=Invoice recipients
   
   summary_found=3
   summary_missing=1
   missing_questions=Can I export a SOC 2 audit report?

   The unrelated password-reset match and low score identify the SOC 2 question as the missing content candidate rather than a marginally covered question.