How to convert text files to documents in Haystack

Search and retrieval components cannot work directly with a folder of plain-text runbooks or policies. Haystack uses Document objects as the handoff format, pairing each file's content with metadata that later pipeline components can clean, split, filter, or store.

The TextFileToDocument converter accepts file paths or ByteStream objects and returns the converted items under the documents output. It reads UTF-8 by default, keeps only the source file name in document metadata, and can attach one metadata dictionary to each source.

Two local fixtures expose the converter boundary without requiring a document store or model. Fail-capable assertions check the document count, metadata order, and decoded content before the script prints values that can be passed to an indexing pipeline.

1. Create the two UTF-8 fixtures under source_docs.

   source_docs/security-runbook.txt

   Reset password requests require MFA confirmation.
   Escalate suspicious sign-ins to security.

   source_docs/billing-policy.txt

   Invoice copies are stored in the billing portal.
   Archive paid invoices after month end.

2. Start convert_text_files.py with the imports and ordered source paths.

   convert_text_files.py

   from pathlib import Path
    
   from haystack.components.converters import TextFileToDocument
    
    
   source_dir = Path("source_docs")
   sources = [
       source_dir / "security-runbook.txt",
       source_dir / "billing-policy.txt",
   ]

3. Add the converter call and per-source metadata after the sources list.

   Append to convert_text_files.py

   converter = TextFileToDocument()
   result = converter.run(
       sources=sources,
       meta=[
           {"team": "support"},
           {"team": "finance"},
       ],
   )
   documents = result["documents"]

   The metadata list must follow the same order as sources. A single dictionary applies the same metadata to every converted document.

4. Append fail-capable assertions and printed evidence to convert_text_files.py.

   Append to convert_text_files.py

   assert len(documents) == 2
   assert [document.meta["team"] for document in documents] == ["support", "finance"]
   assert documents[0].content.startswith("Reset password requests")
   assert documents[1].content.startswith("Invoice copies")
    
   print(f"documents converted: {len(documents)}")
   for document in documents:
       first_line = document.content.splitlines()[0]
       print(f"{document.meta['file_path']} | {document.meta['team']} | {first_line}")
   print(f"result keys: {list(result.keys())}")

   The encoding argument handles text files that are not UTF-8. The store_full_path argument retains the complete source path when downstream components need more than the base file name.

5. Run the completed converter from the Python environment containing haystack-ai.

   $ python convert_text_files.py
   documents converted: 2
   security-runbook.txt | support | Reset password requests require MFA confirmation.
   billing-policy.txt | finance | Invoice copies are stored in the billing portal.
   result keys: ['documents']

   The command exits before printing if conversion drops a file, reorders the metadata, or decodes unexpected content. The two output rows prove that both Document objects retain their source names, team labels, and first content lines.
   Related: How to write documents to a Haystack document store