How to connect LlamaIndex to a llama.cpp server

Local GGUF inference is easier to debug when application code can reach the model server before retrieval, agents, or prompt templates are added. LlamaIndex can send a direct LLM request to a running llama.cpp llama-server process through the server's OpenAI-compatible API.

The OpenAILike integration uses the OpenAI client shape while api_base points at the local /v1 API root. A llama-server alias such as local-llama keeps the model name short and avoids exposing the GGUF file path returned by default.

Start with llama-server already running and reachable from the Python process that runs LlamaIndex. Keep real API keys out of source if the server was started with --api-key, and use a short completion prompt so endpoint or model-name errors fail before larger RAG code is involved.

1. Check the model endpoint exposed by llama-server.

   $ curl http://127.0.0.1:8080/v1/models
   {"object":"list","data":[{"id":"local-llama","object":"model","owned_by":"llamacpp"}]}

   The id value should match the alias or model identifier accepted by the server. Starting llama-server with --alias local-llama keeps this value stable.

2. Install the OpenAI-compatible LlamaIndex LLM integration in the active Python environment.

   $ python3 -m pip install --upgrade llama-index-llms-openai-like
   Successfully installed llama-index-core-0.14.23 llama-index-llms-openai-like-0.7.2 openai-2.44.0

   Use the same virtual environment or deployment image that runs the application.
   Related: How to install LlamaIndex with pip

3. Set the llama.cpp API root for the current shell.

   $ export LLAMA_CPP_BASE_URL=http://127.0.0.1:8080/v1

4. Set the model name returned by the server.

   $ export LLAMA_CPP_MODEL=local-llama

   Replace local-llama with the id from /v1/models when the server uses a different alias.

5. Create a LlamaIndex smoke-test script.

   $ $EDITOR llamaindex-llama-cpp.py

   llamaindex-llama-cpp.py

   import os
    
   from llama_index.llms.openai_like import OpenAILike
    
   base_url = os.environ["LLAMA_CPP_BASE_URL"]
   model = os.environ["LLAMA_CPP_MODEL"]
    
   llm = OpenAILike(
       model=model,
       api_base=base_url,
       api_key=os.getenv("LLAMA_CPP_API_KEY", "not-needed"),
       is_chat_model=False,
       is_function_calling_model=False,
       context_window=4096,
       max_tokens=64,
       temperature=0,
       timeout=120,
   )
    
   response = llm.complete(
       "Write one short sentence confirming LlamaIndex reached llama.cpp."
   )
    
   print(f"endpoint={base_url}")
   print(f"model={model}")
   print(f"response={str(response).strip()}")

   is_chat_model=False uses the OpenAI-compatible completions route. Use is_chat_model=True only when the served model has a chat template and the application should call /v1/chat/completions.

6. Run the script and confirm that LlamaIndex receives model text from llama.cpp.

   $ python3 llamaindex-llama-cpp.py
   endpoint=http://127.0.0.1:8080/v1
   model=local-llama
   response=LlamaIndex reached the llama.cpp server.

   The response wording depends on the loaded GGUF model. The endpoint and model lines should match the exported values, and the response line should contain generated text rather than an HTTP or model-not-found error.

7. Remove the temporary smoke-test script after copying the working settings into the application.

   $ rm llamaindex-llama-cpp.py