How to connect LangChain to a llama.cpp chat server

Local chat models fit into LangChain when the model server speaks a provider API that LangChain already knows how to call. llama.cpp can expose a GGUF model through an OpenAI-compatible chat endpoint, which lets a Python app test local inference without sending prompts to a hosted provider.

The current LangChain path uses the OpenAI chat integration with a custom base_url ending in /v1. The model name should match the llama-server alias or the model ID returned by /v1/models, and sk-no-key-required is enough for a local server that does not enforce API keys.

Keep the llama.cpp server running in a separate terminal while the Python script runs. A short fixed prompt is enough for the smoke test because the important signal is the returned AIMessage and response text, not a specific public-model answer.

Steps to connect LangChain to a llama.cpp chat server:

Start the llama.cpp server with a stable API alias in a separate terminal.
```
$ llama-server -m ./models/local-chat.gguf --alias local-chat --host 127.0.0.1 --port 8080
```
Use a GGUF chat model that has a suitable chat template. The --alias value becomes the model name sent by LangChain.
Check the OpenAI-compatible model endpoint from another terminal.
```
$ curl --silent http://127.0.0.1:8080/v1/models
{"object": "list", "data": [{"id": "local-chat", "object": "model", "created": 1783353600, "owned_by": "llamacpp"}]}
```
If the id is a model path instead of local-chat, use that returned value as LLAMA_CPP_MODEL or restart llama-server with --alias.
Open an activated Python project environment.

Related: How to create a virtual environment for LangChain
Install LangChain and the OpenAI provider integration.
```
$ python3 -m pip install --upgrade langchain langchain-openai
```
langchain-openai supplies the OpenAI-compatible chat client used by init_chat_model.

Create the LangChain smoke-test script.

$ cat > connect_llama_cpp_chat.py <<'PY'
import os

from langchain.chat_models import init_chat_model


model = init_chat_model(
    model=os.environ.get("LLAMA_CPP_MODEL", "local-chat"),
    model_provider="openai",
    base_url=os.environ.get("LLAMA_CPP_BASE_URL", "http://127.0.0.1:8080/v1"),
    api_key=os.environ.get("LLAMA_CPP_API_KEY", "sk-no-key-required"),
    temperature=0,
    max_tokens=64,
    max_retries=0,
)

response = model.invoke("Reply with exactly: llama.cpp chat server reached.")

print(type(response).__name__)
print(response.text)
PY

Set LLAMA_CPP_BASE_URL, LLAMA_CPP_MODEL, or LLAMA_CPP_API_KEY only when the server address, alias, or local API-key policy differs. Keep /v1 at the end of the base URL.

Run the smoke-test script.
```
$ python3 connect_llama_cpp_chat.py
AIMessage
llama.cpp chat server reached.
```
A local model may answer with different wording. The connection is working when the script returns an AIMessage and response text instead of a connection, model-name, or authentication error.
Remove the temporary script after the smoke test.
```
$ rm connect_llama_cpp_chat.py
```