How to connect LangChain to a llama.cpp chat server

Local chat models fit into LangChain when the model server speaks a provider API that LangChain already knows how to call. llama.cpp can expose a GGUF model through an OpenAI-compatible chat endpoint, which lets a Python app test local inference without sending prompts to a hosted provider.

The current LangChain path uses the OpenAI chat integration with a custom base_url ending in /v1. The model name should match the llama-server alias or the model ID returned by /v1/models, and sk-no-key-required is enough for a local server that does not enforce API keys.

Keep the llama.cpp server running in a separate terminal while the Python script runs. A short fixed prompt is enough for the smoke test because the important signal is the returned AIMessage and response text, not a specific public-model answer.

1. Start the llama.cpp server with a stable API alias in a separate terminal.

   $ llama-server -m ./models/local-chat.gguf --alias local-chat --host 127.0.0.1 --port 8080

   Use a GGUF chat model that has a suitable chat template. The --alias value becomes the model name sent by LangChain.

2. Check the OpenAI-compatible model endpoint from another terminal.

   $ curl --silent http://127.0.0.1:8080/v1/models
   {"object": "list", "data": [{"id": "local-chat", "object": "model", "created": 1783353600, "owned_by": "llamacpp"}]}

   If the id is a model path instead of local-chat, use that returned value as LLAMA_CPP_MODEL or restart llama-server with --alias.

3. Open an activated Python project environment.

   Related: How to create a virtual environment for LangChain

4. Install LangChain and the OpenAI provider integration.

   $ python3 -m pip install --upgrade langchain langchain-openai

   langchain-openai supplies the OpenAI-compatible chat client used by init_chat_model.

5. Create the LangChain smoke-test script.

   $ cat > connect_llama_cpp_chat.py <<'PY'
   import os
   
   from langchain.chat_models import init_chat_model
   
   
   model = init_chat_model(
       model=os.environ.get("LLAMA_CPP_MODEL", "local-chat"),
       model_provider="openai",
       base_url=os.environ.get("LLAMA_CPP_BASE_URL", "http://127.0.0.1:8080/v1"),
       api_key=os.environ.get("LLAMA_CPP_API_KEY", "sk-no-key-required"),
       temperature=0,
       max_tokens=64,
       max_retries=0,
   )
   
   response = model.invoke("Reply with exactly: llama.cpp chat server reached.")
   
   print(type(response).__name__)
   print(response.text)
   PY

   Set LLAMA_CPP_BASE_URL, LLAMA_CPP_MODEL, or LLAMA_CPP_API_KEY only when the server address, alias, or local API-key policy differs. Keep /v1 at the end of the base URL.

6. Run the smoke-test script.

   $ python3 connect_llama_cpp_chat.py
   AIMessage
   llama.cpp chat server reached.

   A local model may answer with different wording. The connection is working when the script returns an AIMessage and response text instead of a connection, model-name, or authentication error.

7. Remove the temporary script after the smoke test.

   $ rm connect_llama_cpp_chat.py