How to configure a document preprocessor in Haystack

An indexing pipeline works best when each document reaches the embedder as focused text with enough neighboring context to preserve meaning. Haystack provides DocumentPreprocessor to create those chunks and remove predictable noise in one component.

Haystack runs DocumentSplitter before DocumentCleaner inside DocumentPreprocessor. With passage splitting, split_length counts paragraphs separated by blank lines, while split_overlap repeats a selected number of passages between adjacent chunks before cleanup is applied.

Source lineage still matters after the text changes shape because downstream writers and retrievers must trace each chunk to its original document. Haystack copies existing metadata to every chunk and adds source_id with the input document ID.

1. Create document_preprocessor_demo.py with the Haystack imports and a representative source document.

   document_preprocessor_demo.py

   from haystack import Document
   from haystack.components.preprocessors import DocumentPreprocessor
    
    
   raw_document = Document(
       id="policy-draft",
       content=(
           "CONFIDENTIAL Solar policy drafts need precise source text.\n\n"
           "Retrieval chunks should stay focused on one section.\n\n"
           "Reviewers need overlap when a boundary carries context.\n\n"
           "Source metadata must remain attached for later indexing."
       ),
       meta={"source": "policy-draft.txt"},
   )

2. Append the passage-splitting and cleanup configuration after the source document.

   preprocessor = DocumentPreprocessor(
       split_by="passage",
       split_length=2,
       split_overlap=1,
       remove_substrings=["CONFIDENTIAL "],
       remove_empty_lines=True,
       remove_extra_whitespaces=True,
   )

   Each chunk contains two passages and repeats one boundary passage in its neighbor. Cleanup removes the fixed label and collapses extra whitespace after splitting.

3. Append the component run below the preprocessor configuration.

   chunks = preprocessor.run(documents=[raw_document])["documents"]

4. Append the preprocessing assertions below the returned chunk list.

   assert len(chunks) == 3
   assert all("CONFIDENTIAL" not in chunk.content for chunk in chunks)
   assert all(chunk.meta["source"] == "policy-draft.txt" for chunk in chunks)
   assert all(chunk.meta["source_id"] == "policy-draft" for chunk in chunks)
   assert chunks[0].content.endswith(
       "Retrieval chunks should stay focused on one section."
   )
   assert chunks[1].content.startswith(
       "Retrieval chunks should stay focused on one section."
   )
   assert chunks[1].content.endswith(
       "Reviewers need overlap when a boundary carries context."
   )
   assert chunks[2].content.startswith(
       "Reviewers need overlap when a boundary carries context."
   )

   Execution stops if the component returns the wrong number of chunks, leaves the label in place, loses source metadata, or fails to repeat either boundary passage.

5. Append the chunk display loop after the assertions.

   for index, chunk in enumerate(chunks, start=1):
       print(f"{index}: {chunk.content}")
       print(f"   source: {chunk.meta['source']}")
       print(f"   source_id: {chunk.meta['source_id']}")

6. Verify the configured preprocessor by running the completed program in the haystack-ai environment.

   $ python document_preprocessor_demo.py
   1: Solar policy drafts need precise source text. Retrieval chunks should stay focused on one section.
      source: policy-draft.txt
      source_id: policy-draft
   2: Retrieval chunks should stay focused on one section. Reviewers need overlap when a boundary carries context.
      source: policy-draft.txt
      source_id: policy-draft
   3: Reviewers need overlap when a boundary carries context. Source metadata must remain attached for later indexing.
      source: policy-draft.txt
      source_id: policy-draft

   The repeated first and last passages in neighboring chunks show the configured overlap, while the missing CONFIDENTIAL label and matching source fields show that cleanup and lineage both survived preprocessing.
   Related: How to write documents with DocumentWriter in Haystack