How to configure an OpenAI-compatible chat generator in Haystack

Many model providers, gateways, and local inference servers expose the OpenAI Chat Completions request shape without hosting OpenAI models. Haystack can address one of these services through OpenAIChatGenerator while the surrounding application continues to exchange role-based ChatMessage objects.

The OpenAIChatGenerator component passes requests through the OpenAI Python client. Its api_base_url value identifies the compatible API root, and the client adds /chat/completions when it sends a chat request.

The provider key belongs outside the Python source, while the model identifier must match one advertised by the endpoint. A request-specific identifier returned in an assistant-role ChatMessage confirms the configured request and response path; it does not measure the provider model's answer quality.

Steps to configure an OpenAI-compatible chat generator in Haystack:

Read the provider API key into a hidden shell variable.
```
$ read -rsp "Provider API key: " OPENAI_API_KEY
Provider API key:
```
The hidden input prevents the bearer credential from appearing in the command line, terminal output, or shell history.

Related: How to use API key secrets in Haystack
Export the API key variable to programs started from the current shell.
```
$ export OPENAI_API_KEY
```
Export the provider's versioned Chat Completions API root.
```
$ export OPENAI_BASE_URL="https://llm-gateway.example.com/v1"
```
The provider-supplied value is the API root without an added /chat/completions path.
Export the provider's model identifier.
```
$ export OPENAI_MODEL="acme-chat-small"
```

Create the script foundation with environment-backed configuration values.

$ cat > openai_compatible_chat.py <<'PY'
import os
 
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage
from haystack.utils import Secret
 
api_base_url = os.environ["OPENAI_BASE_URL"]
model = os.environ["OPENAI_MODEL"]
identifier = "cedar-river-47"
PY

Append the configured OpenAIChatGenerator to the script.

$ cat >> openai_compatible_chat.py <<'PY'
 
generator = OpenAIChatGenerator(
    api_key=Secret.from_env_var("OPENAI_API_KEY"),
    api_base_url=api_base_url,
    model=model,
)
PY

Append the system and user chat messages to the script.

$ cat >> openai_compatible_chat.py <<'PY'
 
messages = [
    ChatMessage.from_system("Return only the identifier requested by the user."),
    ChatMessage.from_user(
        f"Repeat this request identifier exactly: {identifier}"
    ),
]
PY

Append the generator call plus reply inspection to the script.

$ cat >> openai_compatible_chat.py <<'PY'
 
response = generator.run(messages=messages)
reply = response["replies"][0]
 
if reply.role.value != "assistant" or reply.text.strip() != identifier:
    raise RuntimeError("The endpoint did not return the requested identifier")
 
print(f"role: {reply.role.value}")
print(f"model: {reply.meta.get('model', 'not reported')}")
print(f"reply: {reply.text}")
PY

Run the completed script to confirm the endpoint returns the request-specific identifier through Haystack.
```
$ python3 openai_compatible_chat.py
role: assistant
model: acme-chat-small
reply: cedar-river-47
```
The matching identifier proves that the compatible endpoint accepted the configured request and that Haystack mapped its response into an assistant ChatMessage. It does not prove how a provider model generated the text.