How to use metadata filters in Haystack

Shared document stores often mix content from several tenants, categories, or sources even though each search must stay inside one allowed slice. Haystack metadata filters narrow the retriever's search space before ranking, so a relevant document outside the requested boundary does not enter the result set.

Haystack expresses each comparison with field, operator, and value keys. A logical dictionary combines comparisons through an AND, OR, or NOT operator, while the supported operators can differ between document-store integrations.

The same runtime filter can be passed directly to InMemoryBM25Retriever.run() or under the retriever component name in Pipeline.run(). The sample program keeps an unfiltered query as a control, then requires both filtered paths to return only the Northwind security document.

1. Create metadata_filter_demo.py with the imports, documents, and in-memory document store.

   metadata_filter_demo.py

   from haystack import Document, Pipeline
   from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
   from haystack.document_stores.in_memory import InMemoryDocumentStore
    
    
   documents = [
       Document(
           content="Password reset requests for Northwind require MFA approval.",
           meta={"tenant": "northwind", "category": "security"},
       ),
       Document(
           content="Password reset requests for Globex use manager approval.",
           meta={"tenant": "globex", "category": "security"},
       ),
       Document(
           content="Invoice export requests for Northwind go through billing.",
           meta={"tenant": "northwind", "category": "billing"},
       ),
   ]
    
   document_store = InMemoryDocumentStore()
   document_store.write_documents(documents)

2. Append the password-reset query and Northwind security filter after the document-store write call.

   query = "password reset"
   filters = {
       "operator": "AND",
       "conditions": [
           {"field": "meta.tenant", "operator": "==", "value": "northwind"},
           {"field": "meta.category", "operator": "==", "value": "security"},
       ],
   }

   Metadata keys use the meta.tenant and meta.category field paths because both values live in each document's metadata dictionary.

3. Append the BM25 retriever and its control and filtered queries below the filter dictionary.

   retriever = InMemoryBM25Retriever(document_store=document_store, top_k=5)
   unfiltered_matches = retriever.run(query=query)["documents"]
   filtered_matches = retriever.run(query=query, filters=filters)["documents"]

4. Append the pipeline query and result checks below the direct retriever calls.

   pipeline = Pipeline()
   pipeline.add_component("retriever", retriever)
   pipeline_matches = pipeline.run(
       data={"retriever": {"query": query, "filters": filters}}
   )["retriever"]["documents"]
    
    
   def tenants(matches):
       return sorted({document.meta["tenant"] for document in matches})
    
    
   print(f"unfiltered tenants: {', '.join(tenants(unfiltered_matches))}")
   print(f"direct filtered tenants: {', '.join(tenants(filtered_matches))}")
   print(f"pipeline filtered tenants: {', '.join(tenants(pipeline_matches))}")
   print(f"filtered content: {filtered_matches[0].content}")
    
   assert tenants(unfiltered_matches) == ["globex", "northwind"]
   assert tenants(filtered_matches) == ["northwind"]
   assert tenants(pipeline_matches) == ["northwind"]

   The unfiltered tenant list proves that both password-reset documents are retrievable. The direct and pipeline assertions fail unless the metadata filter excludes Globex from both filtered paths.

5. Run the completed program to confirm direct and pipeline retrieval exclude the Globex document.

   $ python3 metadata_filter_demo.py
   unfiltered tenants: globex, northwind
   direct filtered tenants: northwind
   pipeline filtered tenants: northwind
   filtered content: Password reset requests for Northwind require MFA approval.

   Both filtered tenant lines must contain only northwind, while the control line must contain globex and northwind.