How to enable response streaming in Haystack

Large language model responses can take long enough that a command-line assistant or chat interface appears idle. Haystack can deliver partial response chunks to application code as they arrive while retaining the complete assistant ChatMessage returned by the generator.

The OpenAIChatGenerator accepts a streaming_callback that receives one StreamingChunk at a time. Text chunks expose chunk.content, while chunk.finish_reason marks the end of the provider stream; tool calls and reasoning output use other chunk fields.

The generator reads its credential from OPENAI_API_KEY through a Haystack Secret and streams one candidate by setting n to 1. An optional OPENAI_API_BASE_URL keeps the same program usable with an OpenAI-compatible endpoint, and the final comparison detects missing, reordered, or duplicated text chunks.

1. Open the Python environment where haystack-ai is installed.

   Related: How to install Haystack with pip

2. Create streaming_generator_demo.py with the imports, chunk buffer, and text callback.

   import os
    
   from haystack.components.generators.chat import OpenAIChatGenerator
   from haystack.dataclasses import ChatMessage, StreamingChunk
   from haystack.utils import Secret
    
   received = []
    
    
   def show_chunk(chunk: StreamingChunk) -> None:
       if chunk.content:
           received.append(chunk.content)
           print(chunk.content, end="", flush=True)

3. Configure OpenAIChatGenerator below show_chunk() to send each streamed chunk to the callback.

   generator = OpenAIChatGenerator(
       api_key=Secret.from_env_var("OPENAI_API_KEY"),
       api_base_url=os.getenv("OPENAI_API_BASE_URL"),
       model=os.getenv("OPENAI_MODEL", "gpt-5-mini"),
       streaming_callback=show_chunk,
       generation_kwargs={"n": 1},
   )

   An unset OPENAI_API_BASE_URL uses the default OpenAI endpoint; an OpenAI-compatible provider requires its chat-completions base URL.
   Related: How to configure an OpenAI-compatible chat generator in Haystack

4. Append the request and comparison below the generator to test streamed text against the completed reply.

   print("stream: ", end="")
   result = generator.run(
       messages=[
           ChatMessage.from_user(
               "Explain Haystack response streaming in one short sentence."
           )
       ]
   )
   reply = result["replies"][0]
   streamed_text = "".join(received)
    
   print(f"\nfinal: {reply.text}")
   print(f"match: {streamed_text == reply.text}")

5. Review the completed streaming_generator_demo.py against the consolidated file.

   streaming_generator_demo.py

   import os
    
   from haystack.components.generators.chat import OpenAIChatGenerator
   from haystack.dataclasses import ChatMessage, StreamingChunk
   from haystack.utils import Secret
    
   received = []
    
    
   def show_chunk(chunk: StreamingChunk) -> None:
       if chunk.content:
           received.append(chunk.content)
           print(chunk.content, end="", flush=True)
    
    
   generator = OpenAIChatGenerator(
       api_key=Secret.from_env_var("OPENAI_API_KEY"),
       api_base_url=os.getenv("OPENAI_API_BASE_URL"),
       model=os.getenv("OPENAI_MODEL", "gpt-5-mini"),
       streaming_callback=show_chunk,
       generation_kwargs={"n": 1},
   )
    
   print("stream: ", end="")
   result = generator.run(
       messages=[
           ChatMessage.from_user(
               "Explain Haystack response streaming in one short sentence."
           )
       ]
   )
   reply = result["replies"][0]
   streamed_text = "".join(received)
    
   print(f"\nfinal: {reply.text}")
   print(f"match: {streamed_text == reply.text}")

6. Set OPENAI_API_KEY in the program's environment without placing the key in source or saved terminal output.

   Related: How to use API key secrets in Haystack

7. Run the completed streaming_generator_demo.py program.

   $ python3 streaming_generator_demo.py
   stream: Response streaming sends text chunks before the final reply.
   final: Response streaming sends text chunks before the final reply.
   match: True

8. Check for match: True after the streamed and final text.

   A False value means the callback path dropped, duplicated, or transformed text before the generator assembled the final ChatMessage.